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OVERVIEW 


cross the United States, college graduation rates for low-income students are too low. There are 

many contributing factors: inadequate academic preparation, the cost of college, challenges 

balancing work and school, difficulties that many first-generation students face navigating 
college, and institutional practices that may unintentionally hold students back. A key element of 
the programs that are most effective at helping students stay on track is frequent advising, includ- 
ing reaching out to students who seem to be struggling. In many cases, however, resources limit 
the amount of time advisers can spend with students. Employing technology to improve the staff’s 
ability to provide high-quality advising can be an attractive strategy for institutions looking to make 
system-wide changes. 


Integrated Planning and Advising for Student Success (iPASS) is an initiative funded by the Bill & 
Melinda Gates Foundation to support colleges that seek to incorporate technology into their advis- 
ing and student services. In iPASS, such technology is intended to increase advising’s emphasis on a 
student’s entire college experience, enabling advisers to more easily (1) intervene when students show 
early warning signs of academic and nonacademic challenges, (2) regularly follow up as students 
progress through college, (3) refer students to tutoring and other support services when needed, and 
(4) provide personalized guidance that reflects students’ unique needs. 


To study how technology can support advising redesign, MDRC and the Community College 
Research Center partnered with three institutions already implementing iPASS: California State 
University, Fresno; Montgomery County Community College; and the University of North Carolina 
at Charlotte. The three institutions increased the emphasis on providing timely support, boosted 
their use of advising technologies, and used administrative and communication strategies to increase 
student contact with advisers. The enhancements at all three institutions are being evaluated using 
a randomized controlled trial research design. 


This report shows that the enhancements generally produced only a modestly different experience 
for students in the program group compared with students in the control group, although at one 
college, the enhancements did substantially increase the number of students who had contact with 
an adviser. Consequently, it is not surprising that the enhancements have so far had no discern- 
ible positive effects on students’ academic performance. The findings also highlight the potential 
for unintended consequences. Before the study, each of the institutions had required that certain 
groups of students see an adviser before registering for classes in the next semester. Each institution 
expanded this preregistration requirement to include all students in the study’s program groups, but 
at one institution, the requirement appears to have contributed to a small reduction in earned credits. 


Even though the enhancements have not yet produced clear improvements in students’ academic 
performance, in interviews, some staff members at the institutions indicated that their work in the 
iPASS initiative and their work on the enhancements studied here are important steps toward a 
stronger system to support students and help them succeed. In general, the institutions in this study 
have made progress in making advising technologies and data accessible to students, advisers, and 
faculty — and in experimenting with new strategies to use these technologies in student advising 
— but each also faced challenges. 


It is still early, and there is much to learn about how to use technologies and advising practices to 
drive improvements in student outcomes. Institutional practices are changing; the next step is to 
build on these advances to produce larger changes in the student experience. 
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PREFACE 


ommunity colleges and broad-access four-year institutions make college possible for millions 

of low-income students across the country. Unfortunately, many students face numerous 

academic and nonacademic obstacles to postsecondary success. A growing body of evidence 
shows that strong advising and student support practices can help alleviate some of these obstacles. 
Many institutions, however, have high student-to-adviser ratios, which limit the time advisers can 
spend with students, make it difficult for advisers to provide sufficient academic guidance, and re- 
duce their opportunities to help students take advantage of other support services. To help address 
these challenges, many institutions are adopting new technologies designed to assist advisers and 
students. Colleges are still learning how best to integrate these new technologies, and it is not yet 
clear whether this approach will prove effective. 


Integrated Planning and Advising for Student Success (iPASS) is an initiative funded by the Bill & 
Melinda Gates Foundation to help colleges redesign their student support services with the aid of 
technology. The initiative includes a research component to build knowledge about what works and 
to share lessons from the institutions’ experiences. This report details the efforts of three institutions 
to extend their iPASS work to a larger group of students, with new enhancements. These institu- 
tions worked with MDRC and the Community College Research Center to evaluate the effects of 
the enhancements using a random assignment study. 


The institutions in this project approached the study as a learning endeavor and an opportunity to 
make research-informed design decisions as they expand iPASS practices. iPASS is an ambitious 
initiative, and the early findings presented in this report show that, while the enhancements mod- 
estly changed students’ experiences, they have not yet produced clear positive effects on students’ 
outcomes. At one college, the expansion of a commonly used strategy to require students to see an 
adviser before registering for classes appears to have led to a small negative effect. Each of the institu- 
tions, however, made progress integrating technology and data with advising, getting more students 
in to see advisers, and expanding the content of advising sessions. Each also faced difficulties. The 
findings reinforce that institutional change is challenging and that it often takes time before reform 
efforts are converted into substantial changes in students’ experiences. 


The report describes the diligent and thoughtful work college staff members did to enhance their 
student support services, providing lessons on the use of technology in advising practice. A report 
next year will provide more detail on the qualitative findings outlined in this report. A final report 
will provide longer-term data so colleges can better understand how to incorporate technology into 
their advising practices. 


Gordon L. Berlin 
President, MDRC 
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EXECUTIVE SUMMARY 


here is wide recognition that postsecondary graduation rates for low-income students are too 

low. Multiple factors contribute to poor college outcomes, including inadequate academic 

preparation, the cost of college, challenges balancing work and school, difficulties that many 
first-generation students face navigating college, and institutional practices that may unintention- 
ally hold students back. Colleges and universities are experimenting with new strategies to improve 
student persistence and completion. Frequent, proactive advising has emerged as a key element of 
the most effective programs that help students stay on track to graduation. 


In many cases, however, resources limit the amount of time advisers can spend with students and 
inhibit the delivery of frequent and consistent high-quality advising. Advisers at community col- 
leges and broad-access universities generally have large caseloads (some colleges assign as many as 
1,200 students to a single adviser). Employing technology to improve the staff’s ability to provide 
high-quality advising can be an attractive strategy for institutions looking to make system-wide 
improvements. Theoretically, if colleges use advising technology effectively, advisers can improve 
their efficiency, spending less time on administrative tasks and more time on student support in- 
formed by data. In practice, however, it is not yet clear whether this use of technology and data will 
be enough to measurably improve student outcomes. 


Integrated Planning and Advising for Student Success (iPASS) is an initiative that the Bill & Melinda 
Gates Foundation began funding in 2013 to support colleges incorporating technology into their 
advising and student services. In iPASS, technology is intended to increase advising’s emphasis on a 
student’s entire college experience, enabling advisers to more easily (1) intervene when students show 
early warning signs of academic and nonacademic challenges, (2) regularly follow up as students 
progress through college, (3) refer students to tutoring and other support services when needed, and 
(4) provide personalized guidance that reflects students’ unique needs. 


To continue to understand how technology can support advising redesign, MDRC and the Community 
College Research Center partnered with two broad-access universities and one community college 
to refine and extend the schools’ work under their existing iPASS grants and to study the effects of 
the enhancements on student outcomes. Using a random assignment research design and qualitative 
research, including interviews with students, advisers, and administrators, the study will provide 
rigorous evidence about the effects of the enhancements and help the institutions advance their 
work and inform the field. 


THE STUDY OF ENHANCED iPASS 


The three schools involved in this study present the opportunity to study the intervention in no- 
tably different contexts. California State University, Fresno (Fresno State), which is designated as 
a Hispanic-Serving Institution, and the University of North Carolina at Charlotte (UNCC), which 
serves a large number of transfer students, are both large four-year institutions with graduate schools, 
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serving mostly full-time students. Montgomery County Community College (MCCC) is a two-year 
college in suburban Pennsylvania, and about two-thirds of its students attend part time. 


The project was supported by institutional leaders and entailed collaboration between college staff 
members and researchers to design a two-semester program that extended the institutions’ existing 
iPASS work to certain groups of students with some enhancements. One of the primary approaches was 
to require at least a subset of students to meet with an adviser at least once, enforced by a registration 
hold. The institutions aimed to enhance the quality of that advising session by encouraging advisers 
not only to talk with students about challenges they may be facing, but also to lead a discussion of 
academic and career goals informed by data gathered from technology adopted as part of the iPASS 
initiative. To motivate this type of discussion, each institution designed a college-specific “toolbox,” 
a document that outlines three to four learning outcomes for students focused on information, skills, 
and cognitive development; topics for discussion; and questions for each topic that advisers may 
consider using to engage students.’ The enhancements also increased use of early-alert systems, in 
which faculty members flag students who appear to be struggling in the first part of the semester. 


The study enrolled eligible students through passive consent? and randomly assigned them to a 
program group, which was offered the two-semester enhanced iPASS program, or a control group, 
which received the institution’s typical advising services — a version of unenhanced iPASS. The 
institutions offered the iPASS enhancements for two semesters, and students were enrolled in the 
study in two cohorts: 5,244 students were randomly assigned in the first cohort in spring 2017, and 
2,767 students were randomly assigned in the second cohort in fall 2017, for a total of 8,011 students, 
with 3,760 in the program group and 4,251 in the control group. First-semester findings presented 
here include both cohorts; the second-semester findings are available only for the first cohort. 


IMPLEMENTATION FINDINGS 


The enhancements generally produced only a modestly different student experience for those in the 
program group, compared with what students in the control group experienced, although at Fresno 
State, the enhancements did substantially increase the number of students who had contact with an 
adviser. Each of the institutions also faced challenges with resource constraints and adviser capacity. 


California State University, Fresno 


Fresno State wanted to use early alerts more effectively to help students experiencing academic or 
nonacademic distress and also wanted to use an academic planning technology called MyDegreePlan 
to help students figure out what pathway they wanted to take and how to stay on track. To achieve 


1 Elizabeth Wilcox, “An End to Checklist Thinking: Learning-Centered Advising in Practice,” NACADA 
Clearinghouse (2016), www.nacada.ksu.edu/Resources/Clearinghouse/View-Articles/An-End-to-Checklist- 
Thinking-Learning-Centered-Advising-in-Practice.aspx. 


2. The eligibility criteria varied by institution; details are given in the following sections. “Passive consent” 
means that students who met the study criteria were automatically enrolled. They were notified of their 
enrollment by email and allowed to opt out of having their data collected. 
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these objectives, the university designed enhanced advising practices with four key components: 
training for students on MyDegreePlan; additional early-alert surveys to faculty members who taught 
students in the program group; phone calls and emails from peer mentors; and required advising 
appointments to map students’ education plans, address early alerts, and discuss strategies for stay- 
ing on course to degree completion. Fresno State placed a registration hold on all students in the 
program group to enforce the advising appointments. 


Key findings: 


e Advisers informed students in the program group about MyDegreePlan and held degree planning 
training sessions as intended, but student participation was low. 


e A significantly higher proportion of students in the program group received an early-alert flag 
than did students in the control group (about 35 percent versus 6 percent in their first semester). 


e Peer mentors experienced some challenges reaching students by phone or email. During the first 
semester, they generally focused on reminding students about the advising session. 


e The enhancements resulted in a substantial increase in the number of students who had contact 
with an adviser. Almost all students in the program group, compared with less than half of the 
control group, had contact with an adviser at least once. 


e Advisers described addressing each topic listed in the toolbox, though there was variation in how 
they integrated this tool into their advising practice. 


Montgomery County Community College 


MCCC’s iPASS enhancements targeted a group of students who were generally not receiving the col- 
lege’s existing iPASS services: at-risk continuing students in a degree program who were not already 
required to meet with an adviser. Students eligible for the study were those determined to have a 
“low” or “moderate” likelihood of persisting to the next semester, based on the college’s predictive 
analytics tool. To provide advisers and students with more data on students’ academic status and 
career aspirations, faculty members were asked to complete an additional early-alert survey for 
students in the program group; program group students were asked to self-report academic and 
nonacademic issues that could impede their academic progress; and program group students were 
asked to complete a career assessment, to use in conversations with advisers. The requirement to 
meet with an adviser was enforced through a registration hold. 


Key findings: 


e Advisers disagreed with the risk assessment of the predictive analytics tool and reported concerns 
that some students who seemed to be performing well had been determined to be at risk. 


e Advisers received more early-alert data on students in the program group than on students in the 
control group in the first semester but not in the second semester. 
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e Few students completed the self-report survey, which was not required, and some older students 
and students near graduation found the required career assessment to be time consuming and 
unnecessary. 


e The program increased outreach to students about academic issues, and advisers sent students 
personalized emails about early alerts, but it is unclear whether students read those emails. 


e The program increased the number of advising appointments, but many students did not meet 
with advisers until after the semester ended. 


University of North Carolina at Charlotte 


UNCC’s iPASS enhancement model focused on identifying at-risk students and conducting outreach 
and advising sessions with them. In its effort to provide more at-risk students with enhanced ser- 
vices, the university used registration holds, frequent and sustained communication with students, 
and a toolbox to guide advising sessions. The enhancements were designed to provide more data 
to students and advisers, including early alerts about student progress in key courses: A unique 
component of UNCC’s enhancement model was an alert to students in the program group if they 
were enrolled in a “critical progression” course for their majors, such as Principles of Accounting 
for business majors or General Chemistry for biology majors. The institution asked the instructors 
of these courses to use the early-alert feature in the fourth week of the semester and send either 
warnings or positive feedback. After receiving the Week 4 early alerts, advisers encouraged students 
with early alerts to make an appointment for a face-to-face advising session. When midterm grades 
arrived, advisers would place registration holds on students who received two D or F grades or one 
D or F in a critical progression course. 


Key findings: 

e Students in the program group received considerably more communications from advisers than 
students in the control group. 

e Similar proportions of students in the program and control groups were flagged for early alerts. 


e Except for the Colleges of Business and Liberal Arts and Sciences, all the UNCC colleges that 
participated in the study already used mandatory registration holds that require all students to 
meet with advisers, so the registration hold provided limited contrast. 


e Slightly more students in the program group had contact with an adviser. About 73 percent of the 
program group had contact with an adviser during the first semester of the intervention, compared 
with nearly 69 percent of the control group, a statistically significant difference; in the second 
semester, there was no significant difference. 


¢ Most advisers used some portion of the toolbox and, in doing so, reported slightly more in-depth 
conversations with both groups of students. 
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EARLY FINDINGS ON STUDENT OUTCOMES 


Overall, the enhancements to iPASS have not yet produced discernible positive effects on students’ 
academic performance. 


e At Fresno State, the iPASS enhancements produced no statistically significant effects on students’ 
short-term educational outcomes. 


e At MCCC, the enhancements to iPASS had a slightly negative estimated effect on students’ aca- 
demic progress, with statistically significant reductions of 0.3 credits in both credits attempted and 
credits earned in the first semester. The mechanics of the registration hold may have negatively 
affected enrollment in seven-week courses that began midsemester. If a program group student 
with a registration hold attempted to register for one of these courses, the student would have 
had little time before the add/drop deadline to contact the adviser to remove the hold. After two 
semesters, for the first cohort, students in the program group had earned an estimated 0.55 credits 
fewer than students in the control group. The estimated negative effect on credits attempted in the 
courses for which the add/drop period fell after the registration hold was placed (which includes 
summer courses) is 0.32 credits. 


e At UNCC, the iPASS enhancements produced no statistically significant effects on students’ 
short-term educational outcomes. 


CONCLUSION 


iPASS is an ambitious initiative to integrate technology, data, and advising. The ultimate goal is to 
improve student outcomes through improved advising practices. So far, the enhancements undertaken 
by institutions in this study have not produced clear evidence of improvements in student outcomes. 
This appears to be largely because the enhancements generally produced only a modestly different 
student experience for students in the program group, compared with students in the control group. 


Still, some staff members at the institutions believe that their work in the iPASS initiative and their 
work on the enhancements studied here are important steps toward a stronger system to support 
students and help them succeed. It is clear that more work is needed: Across the three institutions, 
large proportions of students who were identified as being at high risk still earn Ds or Fs, or do not 
persist into subsequent semesters of college. In many ways the experiences of the institutions in this 
study highlight both the opportunities and the challenges that this kind of effort presents. Cross- 
departmental collaboration allowed the institutions to build on existing strengths and implement 
new strategies for many students. Collecting and responding to data about college students during 
the semester, however, remains a challenge. While promising, advanced data analytics carries risks; 
simpler, more transparent solutions still work better in some cases. And policies can have unin- 
tended consequences: Although registration holds can get more students to meet with advisers, the 
institutional and student-specific details are important. 


This report provides a closer look at how three institutions engaged in a careful process to enhance 
and study their iPASS implementations. In addition to rigorous evidence about the impacts of the 
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enhancements, the report describes important lessons from the institutions’ efforts to expand and 
enhance their iPASS reforms. A subsequent report will provide greater detail about the implementa- 
tion of iPASS enhancements, in addition to guidance for practitioners interested in redesigning their 
advising practices using new technologies. A final report will document the effects on longer-term 
student outcomes. 
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Introduction 


here is wide recognition that college graduation rates for low-income students are too low. 

Multiple factors contribute to poor postsecondary outcomes, including inadequate academic 

preparation, the cost of college, challenges balancing work and school, difficulties that many 
first-generation students face navigating college, and institutional practices that may unintention- 
ally hold students back. Colleges and universities are experimenting with new strategies to improve 
student persistence and completion. Frequent, proactive advising has emerged as a key element of 
the most effective programs that help students stay on track to graduation." 


Advising programs associated with higher persistence and graduation rates have a set of common 
characteristics: They put structures in place to encourage advisers to develop long-term relationships 
with students, in which advisers and students meet frequently throughout a student’s time in college, 
often multiple times each month; to intervene with students when they show early signs of academic 
and nonacademic struggle, based on real-time information; and to provide personalized guidance 
and referrals that align with students’ unique needs.” This kind of advising is not commonplace, 
however; it is resource intensive and challenging to scale up for all students.* 


In many cases, resources limit the amount of time advisers have with students and inhibit the delivery 
of frequent and consistent high-quality advising. Advisers at community colleges and broad-access 
universities generally have large caseloads (some colleges assign as many as 1,200 students to a single 
adviser).* Partly as a consequence, many students meet with advisers only at one or two required 
milestones during their academic careers. Time- and resource-constrained advising sessions often 
focus on course registration. Course information and registration assistance are important, but 
many students face additional barriers to success that can arise at any point during the semester.® 
Consequently, institutions are looking for ways to help advisers work with students on the whole 
student experience. 


1 Karp, Kalamkarian, Klempin, and Fletcher (2016); Scrivener et al. (2015). 


2 See, for example, Scrivener et al. (2015); Rolston, Copson, and Gardiner (2017); Sommo, Cullinan, and 
Manno (2018). 


3 Scrivener et al. (2015); Headlam (2018). 
Kalamkarian and Karp (2015). 
5 Karp (2013); Scott-Clayton (2011). 
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The hope is that technology tools may help improve advising and maximize the use of existing 
resources, and many colleges have begun incorporating technology into their student support prac- 
tices. In a national survey of about 1,000 college and university administrators, faculty members, 
and advisers, over 70 percent of respondents reported using advising technology, such as course 
planning tools.® Advising tools, new technology, and real-time data on students’ progress can help 
advisers identify and reach out earlier to students at risk of poor college outcomes and can facilitate 
advising sessions that cover a broad spectrum of important topics: for example, by providing richer 
data about students and helping students make use of other resources, choose majors and careers, 
and graduate on time. 


Integrating technology and advising can also be an attractive strategy for institutions looking to make 
system-wide changes. New software technology, for example, can be deployed and made accessible to 
all advisers, faculty, staff, and students with relatively small marginal costs after the initial invest- 
ment. Theoretically, if colleges use advising technology effectively, advisers can improve efficiency, 
spend less time on administrative tasks, and free up more time for higher quality, data-informed 
student advising. In practice, however, it is not yet clear whether integrating this kind of technology 
and data into student advising will be enough to measurably improve student outcomes. 


THE INTEGRATED PLANNING AND ADVISING FOR 
STUDENT SUCCESS (iPASS) INITIATIVE 


Integrated Planning and Advising for Student Success (iPASS) is an initiative that the Bill & Melinda 
Gates Foundation began funding in 2013 to support colleges incorporating technology into their 
student support services. The initial funding solely supported the deployment of technology, but 
subsequent grants supported both technology and redesign of advising and student support services. 
In iPASS, advising technology is intended to increase the emphasis of advising on a student’s entire 
college experience by enabling advisers to more easily (1) intervene with students proactively when 
they show signs of academic and nonacademic struggle, (2) regularly interact with students through- 
out their time in college, (3) refer students to tutoring and other support services as needed, and 
(4) provide students with personalized strategies for managing the challenges that arise in college. 


Since 2015, various foundations have given additional iPASS grants to colleges to enact this reform. 
(See Box 1.1 for more information on the grant.) The colleges use diverse strategies and technology 
products, so no two colleges’ iPASS work is identical. Many technology tools implemented through 
iPASS fall into one of these three categories: 


e Education planning tools for selecting programs and courses, mapping out plans to complete a 
program, and tracking progress toward completion 


¢ Counseling and coaching tools for improving students’ connections to support services, such as 
mental health services and academic tutoring 


6 Tyton Partners and Babson Survey Research Group (2016). 
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BOX 1.1 
The iPASS Grant 


In 2015, 26 institutions received grants from the Bill & Melinda Gates Foundation and The Leona 
M. and Harry B. Helmsley Charitable Trust to implement Integrated Planning and Advising for 
Student Success (iPASS) reforms. In addition to funding, all iPASS grantees received support 
from EDUCAUSE or Achieving the Dream to support technology integration and redesign of 
advising services over the three-year grant period. EDUCAUSE is a nonprofit association that 
supports postsecondary institutions in their use of information technology. Achieving the Dream 
is a nonprofit organization focused on evidence-based efforts to improve community college 
students’ outcomes. iPASS grantees also had access to professional development opportunities 
through a broad network of institutions, technology vendors, and technical assistance partners. 


Colleges and universities in the iPASS initiative identified individuals across several departments 
(including Institutional Research and Effectiveness, Information Technology, Student Affairs and 
Enrollment Management, and Academic Affairs) to plan, manage, and implement the integration 
of technologies and the advising reform. During the first year, these teams worked with their 
technical assistance partners to acquire new advising technologies, add new functions to existing 
technologies, and design advising procedures to use these tools. During the second and third 
years of the grant, institutions began using the technologies in the context of their revised student 
support approaches. 


¢ Risk targeting technology, such as early-alert and predictive analytics systems, that provide 
advisers, students, and support staff members with emerging indications of students’ academic 
struggles or predict students’ probability of graduation, enabling advisers to connect students 
with appropriate support 


Research on institutional change initiatives in postsecondary education suggests that undertaking 
an institution-wide strategy is challenging, and it takes time to design and implement.’ In iPASS, 
for example, institutions must select new technology, learn how to use it, collect new data, and help 
faculty and advisers integrate different technology into their practices and ultimately change the 
way they interact with students. 


Early, nonexperimental research on the iPASS initiative, led by the Community College Research 
Center (CCRC), suggests that some colleges implementing iPASS have made good progress toward 
integrating technology into their student support practices.® But many staff members at iPASS col- 
leges are still thinking through the best ways to improve advising and student support with the help 
of technology. For instance, for many colleges, questions remain about how to assign advisers, when 
to intervene with students, and how to communicate with students. Additionally, at many iPASS 
colleges, not all technology tools are in place, many are only partially integrated with each other 


7 Grossman et al. (2015); Mayer et al. (2014); Quint, Jaggars, Byndloss, and Magazinnik (2013). 
8 Karp, Kalamkarian, Klempin, and Fletcher (2016). 
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and with the institutions’ previously existing infrastructure, and many staff members are not us- 
ing the tools to the extent necessary for full iPASS implementation. Large-scale changes are still in 
their early stages,° and there is a growing need to understand how to incorporate technology most 
effectively into colleges’ advising practices. 


THE STUDY 


To better understand how technology can support advising redesign, MDRC and CCRC partnered 
with three institutions — California State University, Fresno; Montgomery County Community 
College in Pennsylvania; and the University of North Carolina at Charlotte. The goal of the project 
is to refine and extend the colleges’ existing iPASS work and study its effects on student outcomes. 


The project entailed collaboration between college staff members and researchers to design a two- 
semester program that extended the institutions’ existing iPASS work to a larger group of students 
with some enhancements. For example, if a college’s iPASS work supported the use of a program 
planning tool, the tool was provided to a new subset of students, combined with additional advising 
outreach over two semesters. Qualitative and quantitative data were collected to understand how the 
changes were being implemented and to assess their effects on student outcomes. These data were 
also used to provide feedback to the institutions, and the qualitative data informed adjustments to 
the program over the course of the study period. 


The study used random assignment to determine who would receive the enhancements and to provide 
rigorous evidence about the effects of those enhancements. Because the institutions continued to 
make adjustments over the course of the study period, the study does not evaluate a static program; 
nor does it seek to evaluate iPASS as a whole at each institution. Instead, this study provides causal 
evidence about enhancements and feedback from qualitative research to help the institutions make 
decisions about next steps and to inform the field. 


Eligible students were enrolled in the study through passive consent"? and were randomly assigned to 
either a program group, which was offered the two-semester enhanced iPASS program, or a control 
group, which received the institution’s typical advising services — a version of unenhanced iPASS. 
Students in the program group were offered the iPASS enhancements for two semesters, and two 
cohorts of students from each institution were included in the study sample. The study launched at 
all three institutions in January 2017; Table 1.1 shows the sample size by research group and institu- 
tion. In spring 2017, the first cohort — a total of 5,244 students — were randomly assigned in the 
study. In fall 2017, the second cohort — an additional 2,767 students — were randomly assigned. In 
total, 8,011 students are in the study sample — 3,760 in the program group and 4,251 in the control 


9 Karp, Kalamkarian, Klempin, and Fletcher (2016). 


10 The eligibility criteria for the study at each college are presented in subsequent chapters. Passive consent 
means that students who met the eligibility criteria were automatically enrolled. They were notified of their 
enrollment over email and were allowed to opt out of having their data collected. 
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TABLE 1.1 The Study Sample, by Institution and Cohort 


PROGRAM CONTROL 
INSTITUTION GROUP GROUP TOTAL 


All institutions 


Cohort 2 618 618 1,236 


Sample size 3,760 4,251 8,011 


group. The impact analyses in this report include data from the first cohort’s first semester (spring 
2017) and second semester (fall 2017) and the second cohort’s first semester (fall 2017). 


The key research questions investigated in this report are: 


1. How did the colleges enhance their iPASS reforms? 
2. Were the iPASS enhancements implemented as intended? 


3. Did the iPASS enhancements produce a different experience for students, compared with stan- 
dard iPASS? 


4. Did the enhancements produce short-term gains in student outcomes, compared with standard 
iPASS? 


In addition to rigorous evidence about the impacts of the enhancements, this study provides impor- 
tant lessons from the experiences of these three institutions and their efforts to expand and enhance 
their iPASS reforms. The findings presented here provide more evidence for how to “do iPASS.” 
The detailed stories from three distinct institutions about the challenges and successes associated 
with advising reform may resonate with other colleges considering this work. A subsequent report 
will go into greater detail about the qualitative findings on the institutions’ iPASS enhancements. 
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A final report will document the effects on longer-term student outcomes, using more data from all 
the student cohorts included in this report. 


Data Sources 


This report relies on multiple data sources. These include student, administrator, and adviser in- 
terviews; iPASS program data, such as early-alert data submitted by faculty members and advising 
appointment data; and student records, which include students’ demographics, key outcome measures 
like credit accumulation and semester-to-semester enrollment, and other data that capture students’ 
academic progress. See Appendix A for information on qualitative data collection and Appendix B 
for a full description of all the quantitative data sources. 


THE STRUCTURE OF THIS REPORT 


The remainder of this report describes iPASS and the enhancements implemented at each college, 
presents early findings about the effects of the enhancements, and provides some preliminary con- 
clusions. Chapter 2 describes the iPASS framework and theory in more detail and provides more 
information on the iPASS work leading up to this point. Chapters 3, 4, and 5 describe the design of 
the program enhancements, their implementation, and their effects at each of the three colleges, as 
well as some conclusions and next steps for the colleges. Chapter 6 concludes with lessons from the 
three institutions and implications for the field. 


6 | Integrating Technology and Advising 


iPASS 


THE THEORY BEHIND iPASS 


The objective of the iPASS reform is to improve students’ short- and long-term academic progress, 
as measured by students’ retention in college and degree completion. The iPASS initiative envisions 
colleges facilitating this improvement by providing high-quality advising that will help students 
clarify their goals and make better academic decisions to achieve those goals.’ This logic is depicted 
in Figure 2.1. 


iPASS Activities 


iPASS activities involve sustained support throughout students’ academic tenure, including regular 
communication from advisers to students inquiring about each student’s needs and experiences. 
Advisers and faculty members identify struggling students. For students experiencing academic or 
nonacademic distress, the goal is for advisers to intervene with emotional and motivational support, 
guidance, and appropriate referrals. Moreover, advisers coordinate with other support personnel 
to ensure that students are receiving a coherent and comprehensive intervention. Ideally, advising 
interactions are personalized — students receive advising that is tailored to their unique circum- 
stances and needs, such as family and work commitments. Finally, advisers teach students the skills 
they need to navigate college.” 


At colleges that use iPASS principles, the ideal student advising experience begins with a required 
meeting with an assigned adviser. The student and adviser discuss degree and career goals and 
chart a multisemester course plan for achieving those goals. The student also learns about campus 
services designed to help him or her stay on track and strategies for managing specific academic 
and nonacademic responsibilities. The student regularly receives messages or phone calls from the 
adviser or another member of the support staff; the amount and type of contact varies depending 
on the student’s needs. If the student experiences academic or nonacademic distress, the student 
receives targeted and coordinated communication from the adviser and others. 


1 Achieving the Dream (2019). 
2 Kalamkarian, Boynton, and Lopez (2018). 
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FIGURE 2.1 iPASS Logic Model 
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sessions 
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Using Technology to Enable iPASS Activities 


Advising technologies may help enable the intended practices by increasing efficiency and providing 
data through education planning tools, early-alert systems, predictive analytics, learning manage- 
ment systems, and communication tools. For example, to target interventions to at-risk students, 
advisers need to know when students are experiencing academic or nonacademic distress. Typically, 
advisers make this assessment by interacting with students or looking up their grades in a learning 
management system. Early-alert and predictive analytic tools can make the process of identifying 
at-risk students more efficient and comprehensive. 


Early alerts, predictive analytics, and other risk data can also enable advisers to differentiate inter- 
ventions depending on the nature and degree of need. Advisers can then strategically allocate their 
limited time and capacity to provide the most intensive support to the students who need it the 
most. For example, advisers may differentiate interventions for students depending on the number 
and type of early-alert flags students receive; those who receive multiple flags may be considered at 
higher risk and be required to meet with an adviser, while students with one flag may be sufficiently 
served with outreach from an adviser via email. Moreover, advisers can use communication tools 
to electronically refer students to resources that align with their needs, such as tutoring or mental 
health services. And communication tools make it easier for advisers to send messages with elements 
of personalization, such as the student’s name and information about appropriate campus resources. 


Communication tools can also help advisers and other personnel coordinate with each other when 
intervening with a student so that the student receives a coherent message from across the institution. 
For example, when an adviser uses this technology to refer a student for services like tutoring, not 
only the student but also the tutor or other staff member providing the service can see the referral; 
the tutor can then follow up with the student. The adviser can also see whether and when the student 
acts on the referral and can later inquire about the student’s experience with the service or ask why 
the student did not follow through with it. 


A shared note-taking platform is another common benefit of communication tools; colleagues can 
see each other’s notes about their interactions with students. When meeting with a student who has 
been served before, staff members can draw on the notes to engage in a more personalized dialogue 
with the student. These tools are also designed to make it easier for staff members to align their 
advising with previously offered information and guidance. 


Advisers’ interactions with students can assume a more instructional focus with the support of 
data and functions available from advising technologies. Advisers can use information about a 
student’s predicted probability of persisting and early indications of academic risk to have a more 
comprehensive discussion during the advising session about the student’s progress and strategies 
for achieving academic and career goals.* In addition, advisers can use education planning tools to 
help students clarify what path they want to pursue and map out the courses that students need to 
take each semester to complete a program of study. In some cases, advisers and students can make 


3  Klempin, Grant, and Ramos (2018). 
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multiple plans, allowing students to compare the courses they would need to take to complete a 
degree for different programs. 


These technologies also allow advisers to track students’ progress toward a degree more efficiently. 
For example, some education planning tools notify the student and the adviser when a student at- 
tempts to enroll in a course that is not part of his or her course plan; the adviser can then either 
intervene to get the student back on track or approve the modification. Overall, by enabling advisers 
and students to engage in multisemester program planning and making it easier for students and 
advisers to know when students are off track, information gathered using these tools can motivate 
discussions that can help students attain their long-term academic and career goals. 


While technology can make it easier to realize the objectives of high-quality advising, adopting 
technology-based practices at full scale often requires redesigning advising structures and practices, 
which can be a lengthy, iterative process. One study found that the institutions that were most suc- 
cessful at using technology to change how students experience support reassessed and improved 
their advising structures and practices on several occasions.* The comprehensive student advising 
experience envisioned in the iPASS model requires time, resources, and continual refinement of 
structures and practices to achieve. 


ENHANCING iPASS AS PART OF THE STUDY 


The Community College Research Center (CCRC) and MDRC are studying the implementation and 
impact of certain iPASS practices at 3 of the 26 institutions that received funding in 2015 to engage 
in iPASS work. The 26 institutions were at different stages of designing and scaling technology- 
based advising practices before the award of the grant. California State University, Fresno (Fresno 
State); Montgomery County Community College (MCCC); and the University of North Carolina at 
Charlotte (UNCC) were among the strongest implementers of technology-based practices. All three 
were also interested in further strengthening their advising processes by making enhancements that 
would accelerate their iPASS reforms. 


Fresno State, MCCC, and UNCC each received supplementary funding from the Bill & Melinda 
Gates Foundation to offset study participation costs (such as time for research-related activities) 
and to expand their advising staff if necessary. As part of the study, these institutions also received 
implementation support and coaching beyond what was available to the entire grantee cohort. CCRC 
and MDRC worked with college administrators and advisers to design the enhancements and to 
provide training on the enhancement components and operations to staff members implementing the 
program. The three institutions convened project leaders for additional meetings with the research 
team and participated in network-building activities together. Over the course of the study period, 
the institutions aimed to make adjustments to the design and delivery of the enhancements. CCRC 
and MDRC continued to provide technical assistance through site visits, regular phone calls, and 
feedback on advising materials through the spring 2018 semester. 


4 Karp, Kalamkarian, Klempin, and Fletcher (2016). 
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The next three chapters in this report provide an overview of the technology-based advising prac- 
tices supported by the 2015 iPASS grant as well as the enhancements at each of the three institutions 
included in this study. In general, for this study, all three institutions increased their use of advising 
technology to gather more precise and timely information about students’ academic or career goals 
and any academic or nonacademic factors that might hinder their academic progress. All three 
institutions also implemented targeted support from advisers and other staff members that was 
informed at least in part by data gathered using technologies adopted through iPASS. For example, 
for this study, the institutions expanded use of their early-alert tools to find out how students in 
the program group were performing at an earlier point in the semester. The institutions also imple- 
mented more intensive tactics for engaging flagged students, such as personalized communication 
with information on applicable resources. 


One of the primary ways institutions prompted students to engage with targeted support was by 
requiring at least a subset of students in the program group to meet with an adviser at least once; 
institutions placed registration holds on these students to motivate them to make an advising ap- 
pointment. Moreover, institutions aimed to motivate a high-quality advising session by encouraging 
advisers both to talk about challenges students might be facing and to lead a discussion of academic 
and career goals based on data such as early alerts. For this purpose, each institution designed a 
college-specific “toolbox.” The toolbox outlines three to four learning outcomes for students focused 
on information, skills, and cognitive development; topics for discussion; and questions for each topic 
that advisers may consider using to engage students.® The toolbox also reminds advisers to check 
whether students have early-alert flags or unsatisfactory midterm grades and to use these data, as 
needed, to prompt the discussion of successes or challenges. The toolbox is intended to help advis- 
ers take a pedagogical approach that focuses on teaching students academic and technical skills 
and supports their development of academic and career goals and a plan for achieving those goals. 


5 Wilcox (2016). 
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California State 
University, Fresno 


alifornia State University, Fresno, known as Fresno State, is a large, urban, public four-year 

institution comprising eight colleges and serving about 21,500 undergraduate students, as 

shown in Table 3.1. A majority of students enrolled at Fresno State attend full-time (88 per- 
cent of undergraduates in fall 2016). It is a designated Hispanic-Serving Institution; nearly half the 
students identify as Hispanic. About 20 percent of students are white, 14 percent are Asian, and only 
a small portion are black or African-American. Fresno State serves a large number of students from 
low socioeconomic backgrounds; about 87 percent of full-time, first-time undergraduate students 
at Fresno State in the 2016-2017 school year were awarded any financial aid. While only 15 percent 
of students complete a bachelor’s degree within four years, the graduation rate rises to 52 percent 
within six years. 


This chapter explains Fresno State’s standard advising process (“unenhanced iPASS”) and how it 
was enhanced for the study. It then describes the students in the study sample and reports on the 
implementation of enhanced iPASS: whether it was conducted with fidelity to its design and how the 
experience of students in the program group contrasted with that of students in the control group. 
The chapter concludes with early findings on academic outcomes. 


iPASS AND STANDARD ADVISING AT FRESNO STATE 


Each of Fresno State’s eight colleges has a separate advising office; the University Advising Center 
serves undergraduates who have not declared a major. While some policies and practices vary by 
college, Fresno State requires students to meet with an adviser before registration during their second 
semester of enrollment, and then again after they have achieved a minimum of 75 credits, usually 
during their junior year. Other advising touchpoints are typically not required unless the student 
is on academic probation, which occurs when a student has a grade point average (GPA) lower than 
2.0. Advising appointments at Fresno State are most often scheduled for 30 minutes. Students are 
typically not assigned a specific adviser; they may see a different adviser each time they seek support, 
which may make it difficult for a student and an adviser to build a connection. 
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TABLE 3.1 Institutional Characteristics, 
California State University, Fresno, Fall 2016 


CHARACTERISTIC VALUE 
Overview 

Degree of urbanization Large city 
Level of institution Four-year and graduate 
Open admission policy No 


Fall enrollment 


Total students 24,405 


Outcomes 


Retention rates* (%) 


200% of normal time 60.0 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS) 2016-2017. 


NOTES: “This represents first- to second-year retention rates of first-time students in fall 2016. 
>Completion rates are calculated for a four-year bachelor's degree (4, 6, and 8 years). 
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The university originally implemented technology-based advising practices to offer support sooner 
to students who may be struggling, before they reach credit milestones that trigger a required inter- 
vention. In 2012, the university began using Grades First, a student support software platform that 
allows the university to identify students who are struggling academically by deploying surveys to 
faculty members during the seventh week of the semester that can generate early alerts. Grades First 
also allows advisers to enter notes from advising sessions into the platform, so other advisers can 
check what their colleagues discussed with students in a previous session and build on that guidance, 
even if it is a particular adviser’s first time directly engaging with the student. 


With the 2015 iPASS grant, Fresno State also launched U.Achieve and U.Direct technologies, which 
are intended to ensure that students at all stages of enrollment have a clear sense of what courses 
they need to take to get and stay on track to graduation. U.Achieve is a degree audit tool that allows 
students and advisers to see how courses taken to date apply to the sequence necessary for comple- 
tion of a degree. U.Direct, commonly known at the university as MyDegreePlan, allows students to 
map the courses they need to take for their program of study each semester until graduation. 


ENHANCEMENT DESIGN 


At the start of this study, the university’s advising technologies had not been fully integrated into 
advising practices. Recording of notes in Grades First varied widely by adviser. Faculty response 
rates to university-wide early-alert surveys were low, often less than 25 percent. Consequently, only 
a small subset of students received an automatically generated early-alert notification. Moreover, 
while MyDegreePlan was available to all students, there was no university-wide effort to promote 
awareness of it; generally, students who learned about the tool could use it, but advisers and admin- 
istrators did not reach out to students and orient them to the tool. 


Fresno State aimed to further develop use of advising technologies, specifically by deploying early 
alerts more effectively to identify students experiencing academic or nonacademic distress, and then 
offering support that would be responsive to each student’s unique needs. The university also wanted 
students to have a complete education plan and for advisers to use MyDegreePlan to help students 
figure out what pathway they wanted to take and how to stay on track. 


To achieve these objectives, the university designed enhanced advising practices for initial use with 
the subset of the target population randomly assigned to the program group.’ Enhanced advising for 
students in the program group included general and targeted communications, an offer of training 
for students on MyDegreePlan, and a required advising session. 


Program group students were invited to attend workshops at the start of the semester in which they 
could complete course plans using MyDegreePlan. Students were notified of these workshops via 
email and phone calls from two peer mentors hired each semester specifically for this study. The 


1 The target population and study sample are further defined in the next section. 


Studying Enhancements to Colleges’ iPASS Practices | 15 


workshops were offered in person for the first cohort of program group students and online for the 
second cohort. 


During Weeks 5 and 10 of the semester, Fresno State disseminated two additional early-alert surveys 
via GradesFirst to faculty members who taught students in the program group. These surveys presented 
faculty members with a list of their students in the program group and requested that they identify 
any students who were exhibiting signs of poor performance in class. Flagged students received an 
automated notification from GradesFirst saying that they had been evaluated as being at risk. 


The two peer mentors hired for the study conducted outreach to all students in the program group 
and targeted calls to students seen as at risk. Each peer mentor was assigned approximately half 
the program group (about 300 students across the two cohorts). At the start of the semester, the 
peer mentors were instructed to call and email their assigned students to remind them to schedule 
appointments with advisers. The peer mentors were also instructed to reach out to students who 
received one early alert after each survey. During each round of outreach, the peer mentors were 
asked to call students first; if they were unsuccessful at reaching a student by phone, they were to 
do two follow-ups by email and phone. Students who received multiple flags were expected to need 
more professional intervention, so the university directed advisers to call those students. 


Finally, the university also placed a registration hold on all students in the program group, which 
required them to meet with an adviser for a longer than normal session — one hour — before 
registering for courses for the next semester. The longer session was intended as an opportunity 
for advisers to engage students in a discussion about their pathway to their long-term goals, using 
the advising “toolbox” as a guide. The toolbox reminded advisers to use the risk data about each 
student that had been gathered through the early-alert surveys and peer mentor calls, and to work 
with students to address the academic and nonacademic issues that may have been hindering their 
progress. To facilitate a discussion about students’ progress toward their long-term goals, the toolbox 
also reminded advisers to review students’ MyDegreePlan — which students were expected to have 
completed after attending a MyDegreePlan workshop at the start of the semester. 


Table 3.2 summarizes the iPASS enhancement components and how they compare with the college’s 
standard courses and services (unenhanced iPASS). 


TARGET POPULATION 


Fresno State participated in this study to improve support for students who, at the start of their second 
year of enrollment, had a GPA between 2.0 and 2.9 and had completed more than 15 but fewer than 


2 The peer mentors were third-year undergraduate students at the start of the study. They received training 
in advising technologies and developmental advising practices. They were supervised by the adviser 
overseeing the study. Peer mentors tracked each attempt to reach a student. The adviser reviewed the 
tracking sheet and on several occasions observed the peer mentors while they called and spoke with 
students. 
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TABLE 3.2 Summary of Enhanced and Standard Advising Models, 
California State University, Fresno 


COMPONENT 


Informational 
communication 


ENHANCED ADVISING 


Message welcoming students to semester, 
announcing MyDegreePlan workshops 


STANDARD ADVISING 


No welcome message required, though 
some advisers may send one 


Early-alert messages at Weeks 5, 7, and 
10 informing students of flags in specific 
courses 


Early-alert message at Week 7 informing 
students of flags in specific courses 


Call from peer mentor reminding students 
to sign up for MyDegreePlan workshop 


No peer mentor phone calls regarding 
MyDegreePlan 


Call from peer mentor checking in on 
flagged students 


No peer mentor phone calls to check in 
on flagged students. Email or call from 
Support Net staff for flagged students; 
process and criteria for outreach vary by 
college. 


Call from peer mentor reminding students 
to sign up for advising appointment 


No peer mentor phone calls reminding 
students to sign up for advising 
appointment 


Call from peer mentor reminding students 
about upcoming advising appointment 


Peer mentor phone calls at one of eight 
colleges reminding students about 
upcoming advising appointment 


Advising required only for students who 
are in their freshman year, have reached 75 
credits, or are on academic probation 


Email campaign sent by adviser to schedule 
appointment 


Walk-in or scheduled appointments initiated 
by student 


30- to 45-minute advising session (varies 
by college) 


No adviser toolbox to guide advising 
session 


Training for students on 
MyDegreePlan 


One-hour workshop on MyDegreePlan, with 
opportunity to complete degree plan with 
facilitators 


SOURCE: Kalamkarian, Boynton, and Lopez (2018). 


No targeted in-person training on 
MyDegreePlan 
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75 credits.* These students typically engage with an adviser only if they seek out advising. There are 
no policies or institutionalized practices for proactive outreach to such students, except for those 
enrolled in a few majors that are in high demand, such as nursing, and those in special programs 
like athletics or the Educational Opportunity Program. 


As shown in Table 3.3, students in the program and control groups were similar in terms of gender, 
race, age, educational history, academic performance at the time of study enrollment, Pell Grant 
eligibility, and first-generation status. About 60 percent of the sample are women, and a majority of 
students are Hispanic. Almost all students in the sample obtained a high school diploma or General 
Educational Development (GED) certificate. On average, students in the sample had earned 44 col- 
lege credits and had a GPA around 2.55 at the start of the study. About 90 percent of the sample were 
of traditional student age (19-24). About two-thirds of the students in the sample were eligible for 
federal Pell Grants, and almost half the students were the first in their families to attend college. 
The first cohort of students enrolled at Fresno State in fall 2015, and the second cohort in fall 2016. 


IMPLEMENTATION FIDELITY AND SERVICE CONTRAST 


e Advisers informed program group students about MyDegreePlan and held degree planning 
workshops as intended. However, student participation in the workshops was low. 


During the spring 2017 semester, the university held 10 workshops as intended; over half the program 
group students attended a workshop. In fall 2017, these workshops were offered online instead, to 
reduce the time advisers had to devote to administering them. Less than 20 percent of the second 
cohort of program group students participated in an online training session. Regardless of whether 
students participated in the workshops, however, they opted to wait until meeting with an adviser 
to create a plan, both advisers and students reported. 


e Early-alert progress surveys were sent to faculty members during Weeks 5 and 10. These pro- 
vided lists of students in the program group only and requested that instructors raise flags 
for these students if warranted by their academic progress. 


As shown in Table 3.4, a significantly higher proportion of students in the program group than in 
the control group received an early-alert flag (about 35 percent compared with 6 percent) in the first 
semester. These data suggest that, as intended, the enhancements to the early-alert process helped the 
university identify students in the program group who were showing signs of struggling academically. 


¢ The university experienced some challenges implementing interventions for students who 
received early-alert flags. 


3 The eligibility criteria were adjusted for the second cohort to include students with a GPA between 2.0 and 
Sul. 
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TABLE 3.3 Demographics of the Sample, 
California State University, Fresno 


PROGRAM CONTROL STANDARD 

CHARACTERISTIC GROUP GROUP DIFFERENCE ERROR 
Female (%) 61.4 62.0 -0.6 2.8 
Race/ethnicity (%) 

Hispanic 63.2 61.8 1.4 2.8 

White 12.4 11.9 0.5 1.9 

Black 3.2 2.3 0.8 0.9 

Other 21.2 24.0 -2.7 2.4 
Obtained a high school diploma or GED? (%) 99.7 99.5 0.2 0.4 
College credits earned at study enrollment 44.40 44.29 0.11 0.70 
Grade point average at study enrollment 2.57 2.54 0.03 0.02 
Age (%) 

18 or under 10.9 10.1 0.9 1.8 

19-24 89.1 89.9 -0.9 1.8 

25 or over 0.0 0.0 0.0 0.0 
Eligible for federal Pell Grant (%) 67.8 65.7 2.2 27 
First person in family to attend college (%) 49.1 49.4 -0.2 2.9 
Sample size (total = 1,219) 610 609 


SOURCE: MDRC calculations using demographic data from California State University, Fresno. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample sizes vary because of missing values. 
4A General Educational Development (GED) certificate is a high school equivalency credential. 


Peer mentors were used for students in the program group who received only one flag. The peer 
mentors called students but were often unable to reach them by phone. Peers left messages describ- 
ing the purpose of their calls and providing their contact information. Peer mentors followed up 
with students via email and subsequent calls at least twice after the initial call if the student was 
not available, but few students emailed or called back. In addition, peer mentors expressed more 
nervousness about calling their peers than expected. Consequently, these calls generally focused on 
reminding students about various pieces of the enhancements, such as the advising session, and less 
about discussing potential barriers to success, especially in the beginning. 
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TABLE 3.4 Differences in Early Alerts, 
California State University, Fresno 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Had any early alerts (%) 34.9 5.8 294°" 2.2 
Number of early alerts? 0.57 0.08 0.49*** 0.04 
Sample size (total = 1,219) 610 609 
Second semester 
Had any early alerts (%) 31.5 8.3 23.1*** 3.1 
Number of early alerts* 0.54 0.10 0.44*** 0.06 
Sample size (total = 602) 302 300 


SOURCE: MDRC calculations using early alerts data from California State University, Fresno. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Alerts in spring and summer semesters are combined. 
The second cohort is not included in the second semester analysis. 
4Due to limitations in the data reported, no more than one early alert per student is counted each day. 


When asked about the phone calls from peer mentors, several of the students in the program group 
said they had not received calls or messages. Those who did report receiving calls characterized 
them, as one student put it, “only reminders of just to come in” and meet with an adviser. Overall, 
the university recognized early in the study that the peer mentorship component of the intervention 
needed improvement. 


Advisers were asked to call students who had multiple early-alert notifications. Fewer than 20 stu- 
dents received multiple flags, so this enhancement was seldom applied. 


e The enhancements resulted in a substantial increase in the number of students who had con- 
tact with an adviser. 


Almost all students in the program group had contact with an adviser at least once, as shown in 
Table 3.5. In comparison, less than half the students in the control group had contact with an adviser 
during the same period. Advisers noted that sessions with program group students often lasted for 
an hour or more and at a minimum took 45 minutes, substantially longer than the typical advis- 
ing appointment experienced by students in the control group who opted to meet with an adviser. 
Advisers agreed that the extra time made a detailed discussion about academic and career pathways 
possible, but they cautioned that it would be difficult to implement for more students, given the large 
number of students assigned to each adviser. 
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TABLE 3.5 Differences in Advising Appointments, 
California State University, Fresno 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Had contact with an adviser (%) 96.9 44.4 52.8*** 2.1 
Had in-person contact with an adviser? (%) 66.9 31.0 35.9*** 2.7 
Number of advising contacts® 1.93 0.73 1.20" 0.07 
Number of contacts with adviser? (%) 
0 3.1 55.9 -52.8*** 21 
1 41.6 28.4 13.2*** 2.7 
2-3 46.5 12.9 33.7*** 2.4 
4 or more 8.8 2.8 6.0*** 1.3 
Sample size (total = 1,219) 610 609 
Second semester 
Had contact with an adviser (%) 91.5 48.9 42.6*** 3.2 
Had in-person contact with an adviser? (%) 62.5 34.1 28.4** 3.9 
Number of advising contacts® 1.38 0.72 0.66*** 0.08 
Number of contacts with adviser® (%) 
0 8.5 51.1 -42.6*** 3.2 
1 61.5 32.1 29.3*** 3.9 
2-3 26.4 16.1 10.2*** 3.3 
4 or more 3.7 0.6 3.1e"* 1.2 
Sample size (total = 602) 302 300 


SOURCE: MDRFC calculations using advising data from California State University, Fresno. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by pre-random assignment characteristics of sample members. 

Advising contacts and notes in spring and summer semesters are combined. 

The second cohort is not included in the second semester analysis. 

@Many advisement contact records did not contain data on whether advisement was conducted in person, over the phone, or 
over email. As a result, the percentage of students with in-person contact with an adviser is underreported. 

Due to limitations in the data reported, no more than one advising contact per student is counted each day. 
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e Advisers generally described discussions with students that adhered to the enhancement design. 


All advisers who were interviewed affirmed that when meeting with a student in the program group, 
especially for the first cohort of students, they worked with the student to set up a MyDegreePlan. 
Advisers had discretion on how many semesters they helped students plan ahead; most said they 
worked with students to plan through the subsequent academic year. 


e Advisers described addressing each of the topics listed in the toolbox, though there was varia- 
tion in how they integrated this tool into the advising practice. 


Most described feeling comfortable with the objectives listed in the toolbox after the first few advising 
appointments with program group students and referring to the toolbox as needed for a refresher. 
Others used the toolbox during most advising sessions to ensure that each of the main topics was 
addressed. 


e Advisers noted differences, and some similarities, in their advising sessions with the two 
study groups. 


Advisers met with more of the program group students than control group students and identified two 
main ways advising sessions with program group students differed from sessions with control group 
students. First, the extended length of the advising session coupled with a few of the more detailed 
questions in the advising toolbox allowed them to probe more deeply into students’ academic and 
nonacademic experiences. Second, MyDegreePlan was used for degree-planning purposes primar- 
ily with program group students. Despite these differences, advisers noted that often conversations 
with program and control group students felt similar, because if control group students met with an 
adviser and volunteered details that suggested the need for additional intervention, advisers read- 
ily provided support or made relevant referrals. Moreover, several advisers stated that they often 
integrated a few of their favorite questions from the toolbox into all their advising sessions, so these 
particular questions may have been used for a small proportion of students in the control group. 


e Fresno State used first-semester lessons to improve enhanced iPASS in the second semester. 


After the first semester of the study, the university took steps to learn from its early experiences and 
strengthen the enhancements. One lesson from the first semester of implementation was that peer 
mentors were diligent about calling students but were unsure about how to discuss students’ academic 
experiences. As a result, the calls from peer mentors largely served as reminders for other components 
of the intervention, such as the advising appointments. Between the first and second semesters of 
the intervention, peer mentors received professional development from project leaders, the research 
team, and campus resources to help them feel more comfortable engaging students during phone 
conversations. Peer mentors also received ongoing support and supervision from project leaders. 


In addition, the university revised the initial peer mentor communications to students in the pro- 


gram group. Instead of combining the peer mentor introduction with information about the required 
advising appointment, the new protocol covered the topics in two distinct messages. Project leaders 
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made this change in an effort to be clearer with program group students about the role of the peer 
mentors at the university. 


Fresno State also worked in collaboration with the research team to update the toolbox for use dur- 
ing the second semester. Originally, the toolbox focused on ways to use MyDegreePlan to build a 
new plan. The revised toolbox contained directions on how to use the previous semester’s data from 
MyDegreePlan during advising appointments: for instance, engaging students about changes in 
their academic and career interests and the modifications such changes would necessitate to their 
degree plan. 


EARLY FINDINGS ON STUDENT OUTCOMES 


Overall, the enhancements to iPASS did not have a discernible effect on students’ academic per- 
formance. The enhancements produced some positive and some negative but generally statistically 
insignificant estimated effects on early measures of student outcomes. As shown in Table 3.6, after 
two semesters of the intervention, students who received the enhancements had earned 23.40 credits 
and students in the control group had earned 22.89 credits, a difference of 0.52 credits that is not 
statistically significant. Persistence rates were high in both groups, as expected for continuing stu- 
dents at a four-year university. Most of the first cohort returned for a second semester — 93 percent 
of students in the program group and 94 percent of students in the control group. However, large 
proportions of students in both groups received a D or an F in any class: 67 percent of students who 
received the enhancements for two semesters earned a D or an F in at least one class, compared with 
70 percent of the control group, a difference that is not statistically significant. 


Studying Enhancements to Colleges’ iPASS Practices | 23 


————————————— | 
TABLE 3.6 Differences in Credits Attempted and Earned, 
California State University, Fresno 


PROGRAM CONTROL STANDARD 
OUTCOME GROUP GROUP DIFFERENCE ERROR 


First semester 
Registered in any course (%) 98.5 98.9 -0.4 0.3 


Total credits earned 11.97 11.72 0.25 0.22 
Received a D or F in any course (%) 52.1 49.2 2.9 2.8 
Withdrew from any course (%) 6.9 6.4 0.6 1.4 
Sample size (total = 1,219) 610 609 


Second semester 


Registered in any course (%) 92.6 94.1 -1.4 2.0 
Totalcreditsattempted = «= 1309 1317 © 0.08 0.33 
Totalcrediseamed ss ss—S 15200 12900 022 (| 0.40 
ReceivedaDorFinanycourse(%)  — - 448 8 4.5 a 44 
Withdrew fromanycourse(%) 10.9 0 18 2.4 
Sample size (total = 602) 302 300 


Cumulative semesters 


Total credits attempted 26.77 26.78 -0.01 0.43 
Total credits earned 23.40 22.89 0.52 0.62 
Received a D or F in any course (%) 66.6 70.3 -3.7 3.8 
Withdrew from any course (%) 15.9 13.7 2.2 2.9 
Sample size (total = 602) 302 300 


SOURCE: MDRC calculations using transcript data from California State University, Fresno. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Spring and summer semester transcript records are combined. 
The second cohort is not included in the analysis of the second and cumulative semesters. 
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Montgomery County 
Community College 


ontgomery County Community College (MCCC) is a two-year college with campuses in 

suburban Blue Bell and Pottstown, Pennsylvania.’ As Table 4.1 shows, the college enrolled 

11,480 students in fall 2016. Only about one-third of students were attending full time, a 
much lower percentage than at the four-year institutions in the study. Two-thirds of full-time stu- 
dents were receiving any financial aid, and 28 percent of students were receiving Pell Grants. The 
majority of students were white, 14 percent were black or African-American, and smaller proportions 
were Hispanic and Asian. 


MCCC, like most community colleges, is an open-admission institution, meaning that any student 
with a high school diploma or the equivalent can enroll. Community college students often face 
academic challenges as well as nonacademic ones, like balancing work and school.” They also have 
lower overall rates of persistence and graduation than students at four-year institutions.* At MCCC, 
only 8 percent of students earn a degree or certificate within two years, and only 26 percent do so 
within four years (Table 4.1). About two-thirds of students who enter community colleges are as- 
sessed as being underprepared for college-level work. Traditionally, these students must complete 
noncredit developmental (remedial) education course sequences before enrolling in certain college- 
level, credit-bearing courses; however, this practice has changed in several states.* 


This chapter explains MCCC’s standard advising process and how its work under the iPASS grant 
was enhanced for the study. The chapter then describes the students in the study sample and reports 
on the implementation of enhanced iPASS: whether it was conducted with fidelity to its design and 
how the experience of students in the program group contrasted with that of students in the control 
group. The concluding section presents early findings on academic outcomes. 


MCCC also has a Culinary Arts Institute in Lansdale, Pennsylvania. 


2 Bailey, Jeong, and Cho (2010). Compared with students attending four-year institutions, community college 
students are more likely to work while attending college. See Ma and Baum (2016). 


3 National Student Clearinghouse Research Center (2018). 
Bailey, Jeong, and Cho (2010). 
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TABLE 4.1 Institutional Characteristics, Montgomery 
County Community College, Fall 2016 


CHARACTERISTIC VALUE 
Overview 

Degree of urbanization Large suburb 
Level of institution Two-year 
Open admission policy Yes 


Fall enrollment 


Total students 11,480 


Outcomes 


Retention rates? (%) 


200% of normal time 26.0 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS) 2016-2017. 


NOTES: “This represents first- to second-year retention rates of first-time students in fall 2016. 
®Completion rates are calculated for two-year degrees or certificates. 
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iPASS AND STANDARD ADVISING AT MCCC 


MCCC students are assigned to professional advisers with expertise in their specific majors.° 
Academic advising and student support services are centrally located in “student success centers” 
on each of the college’s campuses. 


Staff members and administrators at MCCC had worked for several years before this study to revamp 
the college’s advising services. As part of this work, the college used previous iPASS grants to adopt 
several technology platforms to support advising: 


¢ Starfish Connect, which advisers and students use to schedule advising appointments and advis- 
ers use to store notes from advising sessions 


e Starfish Early Alert, which faculty members use to manually flag students enrolled in their courses 
who are struggling academically and to report midterm grades 


e Ellucian’s student educational planning tool 


e Civitas Illume, a predictive analytics tool, which assigns students a “risk score” based on their 
predicted likelihood of not persisting to the next semester 


¢ Acustom tool created in partnership with Blackboard that aggregates various data sources to give 
students a snapshot of their academic standing (including their GPA and the number of credits 
they have completed) when they log into their online student portal 


All these tools were available campus-wide at the time of this study and were used to varying degrees. 


Some MCCC students are required to meet with an adviser. At the time this study began, first- 
semester students and those on academic probation had to meet with an assigned adviser before 
registering for courses. This requirement was intended to support students and ensure they engaged 
with the college’s services. It was implemented through a registration hold that prevents students 
from registering for classes (or making any changes to their registration status) until they meet with 
their adviser, after which the adviser manually removes the hold. 


Most continuing students in degree programs (the majority of students) do not have any advising 
requirements and often do not voluntarily seek out advisers; they are thus less likely to engage with 
any advising services or experience the college’s efforts to revamp advising. Advising appointments 
that do occur are scheduled for 30 to 45 minutes and do not follow a specified format. Midway 
through the semester, faculty members are asked to use the early-alert tool to flag struggling stu- 
dents. Students who are flagged receive automated emails informing them that their professors have 
flagged them. Generally, advisers are not required to provide personalized outreach to supplement 
the automated emails. 


5 Undecided students declare a major in liberal studies. 
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ENHANCEMENT DESIGN 


MCCC’s iPASS enhancements were designed to make advisers’ communications and meetings with 
students more frequent, timely, and personalized. Multiple methods were used to provide advisers 
and students with more data on students’ academic status and career aspirations: 


e Faculty members teaching classes that program group students were enrolled in were asked to 
complete an early-alert survey earlier in the semester (in addition to the typical midterm survey). 
As with the midterm survey, advisers and students were automatically notified if an alert had 
been issued for a student. 


e Program group students were surveyed and asked to self-report academic and nonacademic issues 
that could impede their academic progress. 


e Program group students were asked to complete a career assessment to provide the basis for con- 
versations with advisers about their long-term academic and career goals. 


Advisers were asked to use these data to intervene with students over email and in person. They 
were also provided with tools, such as email templates, to help them suggest personalized academic 
strategies (for example, going to a professor’s office hours for additional help) and make referrals to 
program group students with early and midterm alerts, as well as to help them to provide personal- 
ized feedback to students who completed the self-report survey. 


The main in-person intervention was a required, semistructured, 45-minute advising appointment 
during students’ first semester in the program, followed by a 30-minute required appointment for 
students in their second semester who struggled academically in their first semester.® The advising 
sessions were intended to occur before registration opened for the following semester. The requirement 
to meet with advisers was enforced through a registration hold — consistent with how the college 
enforced required appointments for first-semester students and students on probation. Students in 
the program group were to meet with an adviser before registering, and the adviser was to help them 
address any issues and take steps to improve their performance in their current courses.’ Advisers 
were provided with a “toolbox” — a list of open-ended, guiding questions designed by college staff 
members and members of the research team. The toolbox included prompts to aid advisers in ad- 
dressing students’ academic progress, long-term educational and career plans, and any nonacademic 
issues, using the new data collected as part of the enhancement. To support an informed career 
planning conversation, program group students were asked to complete a career assessment before 


6 Students in the first cohort had a required appointment in their second semester if they received a D, F, or 
W (withdrawal) in any class in their first program semester; had a single-term GPA of 3.0 or lower in their 
first program semester; or had a cumulative GPA of 3.0 or lower. Students in the second cohort had a 
required appointment if they received a D, F, or W in any class in their first program semester. 


7 The hold prevented program group students from making any changes to their registration status and 
was placed immediately after random assignment for all first-semester program group students and 
immediately after the first add/drop deadline for a subset of second-semester students. Advisers removed 
the hold immediately following the required appointment. 
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scheduling the required advising appointment and were unable to have the required appointment 
until they finished the assessment. 


Advisers reached out to students periodically, usually by email, about the registration hold and 
encouraged them to schedule their required appointment. Program group students also received 
informational messages (via the college’s learning management system) to inform them of — and 
prompt them to take advantage of — various on-campus resources, like tutoring, child care, and 
scholarship opportunities. 


Table 4.2 summarizes the iPASS enhancement components and how they compare with the college’s 
standard courses and services (unenhanced iPASS). 


TARGET POPULATION 


MCCC’s iPASS enhancements focused on a group of students who were generally not receiving the 
college’s existing iPASS services: at-risk continuing students in a degree program who were not 
already required to meet with an adviser.® Students eligible for the study were determined to have a 
“low” or “moderate” likelihood of persisting to the next semester, based on the predictive analytics 
algorithm the college was using.® Focusing on this group allowed the college to extend its redesigned 
advising services to a larger proportion of its student population, based on data that suggested they 
could benefit from the enhancements. Eligible students were randomly assigned either to the control 
group, who would receive the college’s standard advising services for degree-seeking, continuing 
students, or to the program group, who received the college’s iPASS enhancements.'° 


Table 4.3 indicates that the study sample shares many demographic characteristics of the institu- 
tion’s student population (Table 4.1). More than half the sample are women, and about 56 percent 
of the sample are white. Eighty-six percent of the study sample were attending part-time, which is a 
higher percentage than among MCCC’s general college population. While most of the sample were 
between the ages of 19 and 24, about 40 percent were older. Overall, the measured characteristics 
of the program and control groups were similar at the outset of the study, as is expected in a large- 
scale randomized experiment. 


8 Continuing students who already had advising requirements, like veterans and students on probation, were 
not eligible for the study. 


9 Asmentioned above, the college used Civitas Learning’s Illume predictive analytics tool. The tool’s 
algorithm is proprietary, so the exact variables included are not publicly available. 


10 Since advisers already have large caseloads, only a portion of each adviser’s students were randomly 
assigned to the program group and the total number was capped. See Appendix B for more details. 
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TABLE 4.2 Summary of Enhanced and Standard Advising Models, 


COMPONENT 


Informal communication 


ENHANCED ADVISING 


Message welcoming students to semester, 
requesting completion of self-report survey 


Montgomery County Community College 


STANDARD ADVISING 


No welcome message required, though 
some advisers may send one 


Message instructing students to complete 
MyCareerPlan 


No message about completing 
MyCareerPlan 


Automated early-alert message at Week 
3 informing students of their progress in 
specific courses 


Message requesting completion of second 
self-report survey 


Message wishing students good luck on 
finals 


No good luck or end-of-semester message 
required, though some advisers may send 


Series of “Did you know?” general 
informational messages throughout the 
semester 


Message responding to self-report survey 
results 


Mandatory advising for all students during 
their first program semester and a subset 
of students during their second program 
semester 


Advising not required for students who are 
beyond their first semester at the college 
and not on probation 


Advisers send messages to students to 
schedule appointment 


Walk-in or scheduled appointments initiated 
by student 


30- to 45-minute advising session (varies 
by appointment reason) 


Adviser toolbox with eight core 
performance areas to guide advising 
session 


SOURCE: Kalamkarian, Boynton, and Lopez (2018). 
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No adviser toolbox to guide advising 
session 


TABLE 4.3 Demographics of the Sample, 
Montgomery County Community College 


PROGRAM CONTROL STANDARD 
CHARACTERISTIC GROUP GROUP DIFFERENCE ERROR 
Female (%) 59.9 58.6 1.3 1.8 
Race/ethnicity (%) 
Hispanic 15 7.3 0.3 1.0 
White 55.8 56.5 -0.8 1.8 
Black 18.3 17.8 0.5 1.4 
Other 18.4 18.4 0.0 1.4 
Obtained a high school diploma or GED? (%) 92.1 91.1 1 1.0 
College credits earned at study enrollment 38.51 37.44 1.07 0.99 
Grade point average at study enrollment 2.58 2.60 -0.02 0.03 
Enrollment status” (%) 
Full-time 11.9 11.3 0.6 1.2 
Less than full-time 85.2 85.8 -0.6 1.3 
Not registered 2.9 2.8 0.1 0.6 
Age(%) ERE 
18 or under 4.4 4.6 -0.2 0.8 
19-24 53.8 55.9 -2.1 1.8 
25 or over 41.8 39.5 2.3 1.8 
Eligible for federal Pell Grant (%) 16.4 15.5 0.9 1.3 
First person in family to attend college (%) 33.8 34.1 -0.3 17 
Sample size (total = 2,989) 1,248 1,741 


SOURCE: MDRC calculations using demographic and transcript data from Montgomery County Community College. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Sample sizes vary because of missing values. 

4A General Educational Development (GED) certificate is a high school equivalency credential. 

Students’ enrollment statuses are based on the number of credits they attempted in the first semester after study 
enrollment. All other measures presented in this table are based on pre-study enrollment data. 
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IMPLEMENTATION FIDELITY AND SERVICE CONTRAST 


e Advisers expressed concerns with the predictive analytics software that identified the sample. 


As explained above, the college used its predictive analytics software to identify at-risk students for 
the study. Advisers, however, reported that it was difficult to intervene with some students because 
it was not clear why they had been determined to be at risk. The software assessed some students 
as “at risk” who had high GPAs, were close to graduation, or had already earned degrees but were 
taking additional classes. The mismatch for these students was not discovered until students showed 
up for their appointments, frustrated by the registration hold. Consequently, though the predictive 
analytics system was used to identify the study sample for both cohorts, it was not used to assess 
students’ risk during the second semester of the study. Instead, at-risk status, which prompted a 
required appointment in students’ second program semester, was determined by students’ perfor- 
mance in their first program semester. This was intended to provide advisers with a more tangible 
basis on which to intervene with students. 


e Advisers received more data on students in the program group than on students in the control 
group in the first semester, but not all students completed their assessments. 


Faculty members completed the additional early-alert survey and increased the percentage of students 
who received an early alert indicating concern from 42 percent in the control group to 51 percent in 
the program group in the first semester, as shown in Table 4.4. In the second semester, there was not 
a significant difference in the percentage of students who received an early alert indicating concern. 


Advisers said they enforced the requirement that program group students complete the career assess- 
ment before their required appointment. Advisers reported, however, that some students (including 
nontraditional students) were frustrated by the career assessment requirement: It was time consuming 
and seemed unnecessary to those close to graduation, or who had already defined a career trajec- 
tory. Still, a few advisers mentioned that the career assessment led to productive conversations with 
students. Over the course of both semesters in the study, 50 percent of program group students in 
the first cohort completed the career assessment and met with an adviser for the required appoint- 
ment (not shown). Few program group students, however, completed the preappointment self-report 
survey, which was not required. 


e The program increased outreach to students about academic issues. 


Table 4.4 shows that students in the program group had more than twice as many early alerts indicat- 
ing concern, referrals, or to-do items as students in the control group — 87 percent compared with 
42 percent in the first semester, and 49 percent compared with 23 percent in the second semester. 
Early alerts, referrals, and to-do items all generate automated messages, indicating that students 
in the program group were sent more outreach about their academic progress than students in the 
control group. 


Based on interview and program monitoring data, advisers sent personalized emails to all program 
group students for whom alerts were issued. Advisers reported that these emails included more 
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TABLE 4.4 Differences in Early Alerts, Montgomery County Community College 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Had any early alerts indicating concern (%) 51.3 42.3 9.0*** 1.8 
Had any early alerts indicating concern, referrals, or to-do items (%) 87.1 42.3 44.8*** 1.5 
Number of early alerts indicating concern 1.36 0.85 0.51**" 0.06 
Number of early alerts indicating concern, referrals, or to-do items 2.40 0.85 1.55*** 0.07 
Number of early alerts indicating concern, referrals, or to-do items (%) 
0 12.9 57.7 -44.8*"* 1.5 
1 29.5 22.5 7.0*** 1.6 
2-3 36.4 14.4 22.0*** 1:5 
4 or more 21.1 5.4 15.8*** 1.2 
Sample size (total = 2,989) 1,248 1,741 
Second semester 
Had any early alerts indicating concern (%) 24.4 22.1 2.3 1.8 
Had any early alerts indicating concern, referrals, or to-do items (%) 48.8 22.6 26.2*** 2.0 
Number of early alerts indicating concern 0.65 0.43 0.22*** 0.06 
Number of early alerts indicating concern, referrals, or to-do items 1.60 0.43 aie 0.08 
Number of early alerts indicating concern, referrals, or to-do items (%) 
0 51.2 774 -26.2*** 2.0 
1 5.8 13.0 7A 1.3 
2-3 28.0 7A 20.9*** 1.6 
4 or more 15.0 2.6 12.4** 1.2 
Sample size (total = 2,075) 802 1,273 


SOURCE: MDRC calculations using early alerts data from Montgomery County Community College. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Estimates are weighted to account for differences in random assignment ratios across advisers. 
Kudos, alerts indicating positive feedback, are excluded from the table. 
Alerts in spring and summer semesters are combined. 
The second cohort is not included in the second semester analysis. 
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information than their typical emails to students. Advisers also responded when students replied. 
Advisers provided suggestions, like meeting with professors and attending tutoring, via email to the 
small proportion of students who completed the self-report survey. However, it is unclear whether 
students read the emails: Advisers reported that few students replied, and students reported that 
they did not remember receiving them. Several advisers reported making phone calls to students 
who were unresponsive to email — a time-consuming process. 


¢ The program increased the number of advising appointments; however, students tended to 
meet with advisers later than intended. 


A big obstacle to implementing the enhancements as designed was the timing with which program 
group students scheduled their required appointments. Advising staff members placed the registration 
hold on program group students’ accounts early in the semester as intended, and a higher propor- 
tion of program group students than control group students had contact with an adviser. As Table 
4.5 indicates, about 59 percent of program group students had contact with an adviser in the first 
semester, compared with 38 percent in the control group. In the second semester, about 32 percent 
of program group students had contact with an adviser, compared with about 21 percent of students 
in the control group. However, based on interviews with advisers and program staff members, most 
program group students scheduled their required appointment after the semester had ended. Classes 
at MCCC often do not fill up, and many students register for courses just before the start of the 
upcoming semester. So students may not have felt a sense of urgency during the semester to meet 
with advisers to lift the registration hold. 


Because of the timing of most required appointments, advisers were unable to use early-alert data 
during in-person advising sessions. These data provided real-time information on the ways in which 
students might be struggling, which could have helped advisers intervene with students to make an 
adjustment while they were still in class. 


After the first semester of the study, the college added a new staff person, called a student success 
specialist, to conduct phone and text campaigns with all students in the program group who had 
required advising appointments to encourage them to meet with their adviser during the semester. 
Many students who did not persist to the second semester had exhibited academic or financial 
challenges, and the college hoped that the student success specialist could help more students meet 
with advisers earlier in the semester to address such issues. Advisers could also refer students to the 
student success specialist. 


e Advisers did not describe substantial differences in the content of advising sessions. 


Advisers reported that the required advising appointments tended to emphasize career planning more 
than their usual appointments, but that otherwise, advising sessions with students in the program 
group were similar in content to sessions with students in the control group. Advisers also reported 
that the guiding questions in the toolbox did not differ substantially from what they typically asked 
students; therefore, they used the guiding questions, as prescribed by the toolbox, inconsistently in 
their sessions with program group students. 
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TABLE 4.5 Differences in Advising Appointments, 
Montgomery County Community College 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Had contact with an adviser (%) 58.9 38.4 20.5*** 1.8 
Number of advising contacts 1.03 0.62 0.40*** 0.04 
Number of advising contacts (%) 
0 411 61.6 -20.5*** 1.8 
1 34.6 24.3 10.2*** ag 
2-3 20.2 11.8 8.4°** 1.3 
4 or more 44 2.2 1.8°** 0.6 
Sample size (total = 2,989) 1,248 1,741 
Second semester 
Had contact with an adviser (%) 31.6 20.7 10:9*** 1.9 
Number of advising contacts 0.48 0.30 0.18*** 0.04 
Number of advising contacts (%) 
0 68.4 79.3 -10.9*** 1.9 
1 21.1 14.0 7A 1.7 
2-3 9.2 6.4 2.8** 1.2 
4 or more 1.3 0.3 1.0** 0.4 
Sample size (total = 2,075) 802 1,273 


SOURCE: MDRC calculations using advising data from Montgomery County Community College. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Estimates are weighted to account for differences in random assignment ratios across advisers. 
Advising contacts and notes in spring and summer semesters are combined. 
The second cohort is not included in the second semester analysis. 
Fall 2017 data includes tutoring sessions. 


e Although the college received additional resources to support the program, adviser capacity 
constraints remained an issue. 


The technology component proved difficult and time consuming for advisers. It was difficult both 
to learn to use the new software and to find time to use it. In particular, using mail merge to send 
personalized emails based on the new data proved particularly frustrating, because the process was 
not automated. Advisers also had trouble finding time to meet with all their students. After the first 
semester of implementation, some emails were automated and the overall number of messages was 
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reduced. The length of advising sessions was also reduced for students in their second semester in 
the study in an effort to reduce the time burden on advisers. 


EARLY FINDINGS ON STUDENT OUTCOMES 


Overall, the enhancements to iPASS appear to have had a slightly negative estimated effect on 
students’ academic progress. Table 4.6 shows that, in the first semester, control group students 
attempted 7.54 credits and earned an average of 5.60 credits. Program group students attempted 
7.29 credits and earned 5.31 credits — both differences are statistically significant reductions of 0.3 
credits, after rounding. 


A closer look at the data suggests that the mechanics of the registration hold may have negatively 
affected enrollment in later-term, short-duration courses. Most of MCCC’s courses are 15 weeks 
long, but the college also offers some 7-week courses. For some of these 7-week courses, the add/ 
drop deadline fell in the middle of the semester — after the registration hold was placed on program 
group students’ accounts. If program group students attempted to register for one of these later-term 
courses midsemester, the hold would have prevented them from registering — with limited time 
before the add/drop deadline to take the necessary action to remove the hold. 


After two semesters, students in the program group had earned an estimated 0.53 credits fewer than 
students in the control group. The estimated negative effect on credits attempted in the courses for 
which the add/drop deadline fell after the registration hold was placed (which includes summer 
courses) is 0.32 credits. 


MCCC and other colleges commonly place registration holds on other students for many reasons, 
and the way the registration hold was placed for program group students aligns with the college’s 
typical practice. These results show that in some cases registration holds may have unintended con- 
sequences. Based on this finding, staff members at MCCC removed the hold for all program group 
students near the end of the spring 2018 semester. The college is considering how holds might be 
affecting other students outside of the study. 


Table 4.6 shows that a large proportion of MCCC students in the study did not return for a second 
semester; only about half of the first cohort of students registered for second-semester classes. Since 
they were continuing students, some may have graduated or transferred to four-year institutions." 
Many probably did not. The retention rate for the study sample is slightly below the rate for com- 
munity college students nationally — 62 percent of all first-time college students who started in fall 
2016 were retained at the institution where they initially enrolled.” 


The program did not have any significant effects on persistence rates, or the rates at which students 
received Ds or Fs or withdrew from courses. 


11 The final report will include graduation rates. 
12 National Student Clearinghouse Research Center (2018). 
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TABLE 4.6 Differences in Credits Attempted and Earned, 
Montgomery County Community College 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Registered in any course (%) 97.0 97.5 -0.5 0.4 
Developmental 10.3 9.4 0.9 1 
College-level 95.1 95.9 -0.8 0.6 
Course add/drop deadline was before registration 
hold was applied 96.1 96.9 -0.8 0.5 
Course add/drop deadline was after registration 
hold was applied 28.3 29.5 -1.2 1.7 
Total credits attempted 7.29 7.54 -0.26* 0.15 
Developmental 0.37 0.36 0.01 0.04 
College-level 6.92 718 -0.27* 0.15 
From enrolled courses with an add/drop deadline 
before the registration hold was applied 5.95 6.04 -0.09 0.11 
From enrolled courses with an add/drop deadline 
after the registration hold was applied 1.33 1.50 -0.17* 0.10 
Total credits earned 5.31 5.60 -0.29* 0.16 
Developmental 0.19 0.20 -0.01 0.03 
College-level 5.12 5.40 -0.28* 0.16 
From enrolled courses with an add/drop deadline 
before the registration hold was applied 4.24 4.40 -0.16 0.12 
From enrolled courses with an add/drop deadline 
after the registration hold was applied 1.07 1.20 -0.13 0.09 
Received a D or F in any course (%) 32.8 33.5 -0.7 1.7 
Withdrew from any course (%) 17.0 15.7 1.3 1.4 
Sample size (total = 2,989) 1,248 1,741 
(continued) 
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TABLE 4.6 (continued) 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
Second semester 
Registered in any course (%) 51.9 54.4 -2.6 2.2 
Developmental 2.8 2.6 0.1 0.7 
College-level 51.6 54.3 -2.7 2.2 
Course add/drop deadline was before registration 
hold was applied 50.7 52.2 -1.5 2.2 
Course add/drop deadline was after registration 
hold was applied 9.0 11.7 -2.7** 1.3 
Total credits attempted 3.95 4.12 -0.18 0.19 
Developmental 0.10 0.08 0.02 0.02 
College-level 3.85 4.04 -0.19 0.19 
From enrolled courses with an add/drop deadline 
before the registration hold was applied 3.60 3.67 -0.07 0.18 
From enrolled courses with an add/drop deadline 
after the registration hold was applied 0.34 0.45 -0.11* 0.06 
Total credits earned 2.98 3.16 -0.18 0.17 
Developmental 0.04 0.06 -0.02 0.02 
College-level 2.94 3.10 -0.17 0.17 
From enrolled courses with an add/drop deadline 
before the registration hold was applied 2.72 2.84 -0.12 0.16 
From enrolled courses with an add/drop deadline 
after the registration hold was applied 0.26 0.32 -0.06 0.05 
Received a D or F in any course (%) 15.6 15.9 -0.3 1.6 
Withdrew from any course (%) 9.5 11.3 -1.8 1.3 
Sample size (total = 2,075) 802 1,273 
(continued) 
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TABLE 4.6 (continued) 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
Cumulative semesters 
Total credits attempted 11.70 12.25 -0.55* 0.31 
Developmental 0.50 0.47 0.03 0.06 
College-level 11.20 11.78 -0.59* 0.31 
From enrolled courses with an add/drop deadline 
before the registration hold was applied 9.65 9.88 -0.23 0.24 
From enrolled courses with an add/drop deadline 
after the registration hold was applied 2.05 2.38 -0.32** 0.15 


From enrolled courses with an add/drop deadline 


before the registration hold was applied 6.99 7.31 -0.32 0.25 
From enrolled courses with an add/drop deadline 
after the registration hold was applied 1.67 1.88 -0.21 0.14 
Received a D or F in any course (%) 41.6 421 -0.5 2.1 
Withdrew from any course (%) 24.7 25.2 -0.5 1.9 
Sample size (total = 2,075) 802 1,273 


SOURCE: MDRC calculations using transcript data from Montgomery County Community College. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by pre-random assignment characteristics of sample members. 

Estimates are weighted to account for differences in random assignment ratios across advisers. 

Spring and summer transcript records are combined. 

The second cohort is not included in the analysis of the second and cumulative semesters. 

At all three colleges in this study, registration holds were applied to program group students shortly after they enrolled in 
courses for the semester, to require students to meet with an adviser before making further changes to their course loads. At 
MCCCG, this hold prevented some students from registering later for courses that did not follow the regular 15-week course 


schedule, including summer term courses. 
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CHAPTER 


University of North Carolina 
at Charlotte 


home to approximately 23,400 undergraduate students,’ most of whom attend full-time, as 
well as about 3,000 faculty and staff members. About 60 percent of the students are white, 
and, as Table 5.1 shows, many students who attend UNCC are from low socioeconomic backgrounds, 


/\ large, urban research university, the University of North Carolina at Charlotte (UNCC) is 


with nearly 76 percent of full-time beginning undergraduate students receiving some form of finan- 
cial aid. About one-quarter of students finish a bachelor’s degree within four years. UNCC is made 
up of eight academic colleges, four of which participated in this project: the Colleges of Business, 
Computing and Informatics, and Liberal Arts and Sciences and the University College, which houses 
students who have not yet declared a major. 


This chapter explains UNCC’s standard advising process and how its work under the iPASS grant 
was enhanced for the study. It then describes the students in the study sample and reports on the 
implementation of enhanced iPASS: whether it was conducted with fidelity to its design and how the 
experience of students in the program group contrasted with that of students in the control group. 
The chapter concludes with early findings on academic outcomes. 


iPASS AND STANDARD ADVISING AT UNCC 


UNCC has a decentralized model of advising: Each of the eight colleges has its own advising center. 
While all students are required to meet with an adviser during their freshman year and when declaring 
a major, each of the colleges runs its advising center slightly differently. The Colleges of Computing 
and Informatics and Liberal Arts and Sciences use a “split” model of advising: Professional advisers 
meet with incoming students and those who have not declared a major, and faculty members advise 
students once they have entered a major. In the College of Business, professional advisers handle 
advisement until graduation. 


1 The university has approximately 5,300 graduate students, for a total student population of about 28,700. 
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TABLE 5.1 Institutional Characteristics, University of 
North Carolina at Charlotte, Fall 2016 


CHARACTERISTIC VALUE 
Overview 

Degree of urbanization Large city 
Level of institution Four-year and graduate 
Open admission policy No 


Fall enrollment 


Total students 28,721 


200% of normal time 58.0 


SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated 
Postsecondary Education Data System (IPEDS) 2016-2017. 


NOTES: “This represents first- to second-year retention rates of first-time students in fall 2016. 
Completion rates are calculated for a four-year bachelor's degree (4, 6, and 8 years). 
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Some colleges used registration holds to require certain students to see an adviser while others did 
not. University College and the College of Computing and Informatics required all students to meet 
with an adviser every semester and enforced this policy with a registration hold. The Colleges of 
Business and Liberal Arts and Sciences could not place registration holds on all students because 
they did not have the advising capacity to deal with all of them. Advisers at all colleges use a case 
management system that enables them to take notes, see any flags students receive, and conduct 
appointment campaigns.? As part of standard practice, college advising centers have both scheduled 
and walk-in hours; however, the colleges serve walk-in students in different ways. For instance, some 
colleges allow walk-in students to meet with advisers about lifting a registration hold, while others 
only offer walk-in appointments for basic services (such as withdrawing from a course) and require 
a scheduled appointment to lift a hold. Advisers at all colleges refer students to various support 
services as needed, such as tutoring, counseling, and the career center. 


When UNCC began iPASS work in 2015, the goals were to integrate existing systems, update or replace 
systems that were antiquated, and identify and reach students who were considered at risk of not 
graduating. The institution implemented an early-alert system aimed at supporting such students. 
To use the system, faculty members would log in and flag students who seemed to be struggling. The 
system also enabled instructors to give students positive feedback, for example if they were perform- 
ing well or had good attendance. Students and advisers received notification that a flag had been 
raised. However, there was no systematic or consistent policy or procedure for advisers to respond 
to those flags. Though the early-alert system was implemented across the institution as part of the 
original iPASS grant, faculty usage, communication with students, and adviser involvement varied 
by college. The university also purchased a degree audit tool shortly before the study enhancements 
were launched and was in the process of implementing it during the study. 


ENHANCEMENT DESIGN 


Starting in spring 2017, UNCC implemented a two-semester enhancement model that focused on 
identifying at-risk students and conducting outreach and advising sessions for those students. The 
goals were to extend the institution’s existing iPASS work and provide more at-risk students with 
enhanced services. The overall strategy included registration holds to motivate at-risk students to 
meet with advisers; frequent and sustained communication with students; and the use of a “toolbox” 
to guide advising sessions. The enhancements were designed to provide more data to students and 
advisers, including early alerts about student progress in key courses. Table 5.2 presents a summary 
of the iPASS enhancement components and how they compare with the college’s standard courses 
and services (unenhanced iPASS). 


For UNCC, a unique component of the enhancement model was an alert to program group students 
if they were enrolled in a “critical progression course” — a course considered predictive of success 
in a major, such as Principles of Accounting for a business major or General Chemistry for a biology 


2 Appointment campaigns are mass emails to students encouraging them to make an advising appointment. 
The email contains a link that takes students to a website where they can schedule it. 
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TABLE 5.2 Summary of Enhanced and Standard Advising Models, 
University of North Carolina at Charlotte 


COMPONENT 


Informal communication 


ENHANCED ADVISING 


Message to welcome students and assign 
“advising homework” or intake form 


STANDARD ADVISING 


No welcome message required, though 
some advisers may send one. No “advising 
homework” required. 


Standard messages sent via mail-merge 
application and based on enrollment in 
critical progression courses, early alerts, and 
midterm grades 


No customized messages based on 
students’ risk status 


Targeted communication 


At least five messages from adviser over 
the course of the semester asking students 
to establish goals and providing feedback 
based on academic performance and early 
alerts 


Communication sent to students on a 
case-by-case basis but with no systematic 
outreach process 


Mandatory advising session for students 
who get two or more D or F grades on 
midterms or one D or F on a midterm ina 
critical progression course 


Walk-in or scheduled appointments initiated 
by student 


Adviser toolbox to guide advising session, 
with three overarching guiding questions 
and instructions on how to integrate risk 
information 


SOURCE: Kalamkarian, Boynton, and Lopez (2018). 


No adviser toolbox to guide advising session 


major.* Advisers sent messages about those courses to students in the program group in the second 


week of the semester. The institution asked faculty members who taught certain courses — critical 


progression courses and several others — to use the early-alert feature in the fourth week of the 


semester to send either warnings or positive feedback based on how students were performing. 


A key goal of the enhancements was for students struggling academically to meet face to face with 


advisers to discuss their academic progress during the semester. When faculty submitted early alerts 


during Week 4, advisers reached out to students with two or more early alerts or a single early alert 


in a critical progression course, encouraging them to make an appointment for an advising session. 


During Week 8, when the faculty reported midterm grades, advisers placed registration holds on 


3 Critical progression courses were identified using a predictive analytics software tool. 
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students who received two or more D or F grades or received a D or F ina critical progression course. 
Advisers conducted appointment campaigns to those students and required an in-person meeting, 
after which the adviser would remove the registration hold. These outreach procedures were unique 
to the study and to the program group students. 


During the advising sessions, advisers used the toolbox, which consisted of guiding questions about 
what was going well and what was not going well for the student, and probing questions about things 
like career aspirations and academic and nonacademic challenges. The toolbox also contained stu- 
dent learning outcomes (such as learning how to think critically about their experiences). Advisers 
would use the software to take notes and record significant points for students to take away from 
the session. The overall goal was to enable advisers to better identify students’ challenges (academic 
and nonacademic), discuss ways for them to overcome these challenges, and refer them to other 
resources as appropriate, such as tutoring or the campus writing center. Advisers would counsel 
students on topics like improving study skills, developing a time management plan, preparing for a 
test, and considering alternative education pathways. 


To enhance communication, project team leaders from UNCC and the research teams from CCRC 
and MDRC worked together to develop email message templates and timelines. Students in the 
program group received messages every two weeks until midterms, as well as before and after the 
semester. Between midterms and the end of the semester, advisers would reach out to students who 
were not responsive to previous messages concerning their academic performance. Advisers normally 
send some standard messages to all their students, so students in the control group would receive 
some adviser communication. Students in the program group, however, received several unique 
targeted messages, such as those that alerted them to the critical progression courses in which they 
were enrolled, messages concerning early-alert flags and midterm grades, and some additional 
standardized content (for example, information and links to support services and the online advis- 
ing appointment system). iPASS leaders at the university regularly distributed outreach templates, 
instructions, and descriptions of the messages before each appointment campaign or message to 
advisers. Based on feedback from advisers after the first semester, the project leaders updated the 
language of the email templates for the second semester to be more personalized and instituted more 
options to customize messages. 


Shortly after the end of the semester, advisers sent a message called a “Success Report” to students in 
the program group, which included a summary of students’ grades with indicators of which courses 
were critical progression courses. The report also listed courses in which the student was enrolled 
for the following semester and indicated which ones were critical progression courses. 


TARGET POPULATION 


UNCC’s iPASS enhancements targeted two cohorts of students. The first cohort of students, who 
were randomly assigned in the study in spring 2017 and for whom there are two semesters of data, 
consisted of continuing students with fewer than 60 credits. The goal was to identify students who 
had a GPA above 2.0 but less than a 70 percent chance of graduating based on the predictive analyt- 
ics tool. UNCC, however, experienced an error with its predictive analytics system. Many students 
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received incorrect risk scores and may have been included in the study erroneously. Advisers indi- 
cated that the error made them reluctant to rely on the risk scores to inform their intervention with 
students. Instead they used course-specific grades, GPA, and information gleaned from interactions 
with students. 


For the second cohort of students, who were randomly assigned in fall 2017 and for whom there is 
one semester of data, UNCC focused on transfer students. About 45 percent of the incoming students 
at UNCC each fall are transfer students, a group with certain risk factors: They tend to drop out at 
higher rates and have lower GPAs than continuing students, and they are more likely to be balancing 
school with work responsibilities.* 


Table 5.3 presents demographic characteristics for students at UNCC who were enrolled in the study. 
The demographic characteristics of the study sample closely resemble those of the university 
population (Table 5.1), with a couple of small differences: The study sample was about 43 percent 
female, compared with 47 percent of the total student population, and less than 5 percent of the 
study sample were 25 years old or older, compared with about 15 percent of all undergraduate stu- 
dents at UNCC.® 


IMPLEMENTATION FIDELITY AND SERVICE CONTRAST 


e Students in the program group received considerably more communications from advisers 
than students in the control group. 


Advisers sent messages to program group students every two weeks for the first half of the semester, 
in addition to the infrequent messages they sent to all students. Advisers also sent more detailed, 
personalized information to students in the program group, such as information about critical pro- 
gression courses, or a list of resources that could help students with particular needs. 


¢ Similar proportions of students in the program and control groups were flagged for early alerts. 
As Table 5.4 shows, just under one-third of study participants in both groups received any early alert 


during their first program semester.® Only about 15 percent of students in the first cohort in either 
group received at least one early alert during their second semester in the study. 


Duggan and Pickering (2008). 


5 University statistics from 2017 are from the National Center for Education Statistics database of the 
Institute of Education Sciences (https://nces.ed.gov/). 


6 Alerts indicating positive performance were excluded from this analysis, since they were available only 
for the first cohort of students. UNCC used a different alert data system in fall 2017, which did not include 
positive performance alerts. 
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TABLE 5.3 Demographics of the Sample, 
University of North Carolina at Charlotte 


PROGRAM CONTROL STANDARD 

CHARACTERISTIC GROUP GROUP DIFFERENCE ERROR 
Female (%) 43.7 43.1 0.7 1.6 
Race/ethnicity (%) 

Hispanic 9.1 10.0 -0.8 1.0 

White 59.9 60.3 -0.4 1.6 

Black 14.5 14.4 0.2 a |Pa 

Other 16.5 15.4 11 1.2 
Obtained a high school diploma or GED? (%) 99.5 99.5 -0.1 0.2 
College credits earned at study enrollment (cohort 1 only)? 29.62 29.80 -0.19 0.57 
Grade point average at study enrollment (cohort 1 only)? 3.07 3.04 0.02 0.02 
Age (%) 

18 or under 29.5 28.4 1.2 1.5 

19-24 66.2 66.8 -0.5 i oy 

25 or over 4.2 4.8 -0.6 0.7 
Sample size (total = 3,803) 1,902 1,901 


SOURCE: MDRC calculations using demographic data from the University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Sample sizes vary because of missing values. 
4A General Educational Development (GED) certificate is a high school equivalency credential. 
Since the second iPASS cohort at UNCC included primarily transfer students who had no credits or GPA history with the school, only 
the first cohort is included in the measure. There are 2,567 students in cohort 1, with 1,284 in the program group and 1,283 in the control 


group. 


¢ Slightly more students in the program group had contact with advisers. 


Table 5.5 indicates that about 73 percent of the program group had contact with an adviser during 
the first semester of the intervention, compared with about 69 percent of the control group, an 
estimated difference of 4 percentage points that is statistically significant. Except for the Colleges 
of Business and Liberal Arts and Sciences, the UNCC colleges that participated in the study use 
mandatory registration holds that require all students to meet with advisers, so the registration 
hold itself provided limited contrast. In students’ second semester in the study (data limited to the 
first cohort), there was no statistically significant difference in the percentage of students who had 
contact with an adviser; about half the students in both groups did so. 
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TABLE 5.4 Differences in Early Alerts, 
University of North Carolina at Charlotte 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Had any early alerts (%) 30.3 31.5 -1.2 1.5 


4 or more 0.9 1.0 -0.1 0.3 
Sample size (total = 3,803) 1,902 1,901 


Second semester 
Had any early alerts (%) 15.7 14.4 1.6 1.4 


0 84.3 85.9 -1.6 1.4 

1 12.8 11.8 1.0 1.3 

2-3 2.9 2:2 0.7 0.6 

4 or more 0.0 0.2 -0.1 0.1 
Sample size (total = 2,567) 1,284 1,283 


SOURCE: MDRC calculations using early alerts data from the University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Kudos, alerts indicating positive feedback, are excluded from the table. 
Alerts in spring and summer semesters are combined. 
The second cohort is not included in the second semester analysis. 
*Due to limitations in the data reported, students have no more than one early alert per course counted each day. 


48 | Integrating Technology and Advising 


ESS a es 
TABLE 5.5 Differences in Advising Appointments, 
University of North Carolina at Charlotte 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Had contact with an adviser (%) 72.9 68.9 41°" 1.4 
Number of advising contacts 1.12 1.06 0.06* 0.03 
Number of advising contacts (%) 
0 27.1 31.1 41°" 1.4 
1 46.8 44.6 2.3 1.6 
2-3 23.0 21.2 1.9 1.3 
4 or more 3.1 3.2 -0.1 0.6 
Sample size (total = 3,803) 1,902 1,901 
Second semester 
Had contact with an adviser (%) 51.6 49.7 1.9 2.0 
Number of advising contacts 0.73 0.67 0.05 0.03 
Number of advising contacts (%) 
0 48.4 50.3 -1.9 2.0 
1 35.6 36.4 -0.9 1.9 


4 or more 


Sample size (total = 2,567) 1,284 1,283 


SOURCE: MDRC calculations using advising data from the University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Advising contacts and notes in spring and summer semesters are combined. 
The second cohort is not included in the second semester analysis. 
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¢ Most advisers used some portion of the toolbox and, in doing so, reported slightly more in- 
depth conversations with program group students, but also with other students they advised. 


Though very few advisers reported using the toolbox document on a regular basis, several advisers 
indicated that they had memorized the three main guiding questions and used them with students 
quite frequently. According to advisers, the questions facilitated a slightly more in-depth conversation 
with students, opening the conversation to things like nonacademic issues that may be affecting a 
student’s academic performance. Most advisers who used the guiding questions reported using them 
with all their students, because they believed they were helpful in driving conversations, because the 
questions generally aligned with their approach to advising, or because they did not feel comfortable 
using an approach they believed was useful with some students and not others. 


¢ For advisers who reported not using the toolbox, the reasons varied. 


Some advisers reported that they were already practicing high-quality advising with all students 
and did not find that the toolbox offered additional value. Others considered the outreach strategy, 
not the advising sessions themselves, to be the service contrast between the program and control 
groups, so the toolbox did not strike them as useful. One adviser who was hired after the study 
began reported not having been trained to use the toolbox. A few others noted that they may have 
missed the opportunity to use the toolbox with some of the students in the program group because 
of technical difficulties using the system. All these reasons further limited the service contrast.” 


EARLY FINDINGS ON STUDENT OUTCOMES 


At UNCC, the iPASS enhancements did not produce statistically significant effects on students’ 
short-term educational outcomes, as shown in Table 5.6. During students’ first program semester, 
almost all study participants enrolled in classes. Students in both the program and control groups 
earned about 13 credits in their first semester in the program, on average. About 32 percent of both 
program and control group students received a D or an F in their first program semester, and about 
one-quarter withdrew from at least one class. 


During the second program semester, nearly 90 percent of students in the first cohort enrolled in 
classes. Program and control group students both earned an average of about 11.5 credits in their 
second semester in the study, and about 26 percent of program and control group students received 
aD or Fina class in their second semester in the study. Small differences on these measures are not 
statistically significant. 


7 The limited use of the toolbox may also have been a byproduct of a technology change that took place 
during summer 2017. While advisers generally reported liking the new case management and early-alert 
system, they described a learning curve, and many advisers reported that they were still getting familiar 
with the system in late fall, making it a challenge to learn to use the toolbox as well. Moreover, the new 
system was not able to house the toolbox document, so it could be difficult for advisers to remember to 
use it during advising sessions. 
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_ | 
TABLE 5.6 Differences in Credits Attempted and Earned, 
University of North Carolina at Charlotte 


PROGRAM CONTROL STANDARD 
STUDENT OUTCOME GROUP GROUP DIFFERENCE ERROR 
First semester 
Registered in any course (%) 98.2 98.3 -0.1 0.3 


Withdrew from any course (%) 23.7 24.2 -0.5 1.4 
Sample size (total = 3,803) 1,902 1,901 


Second semester 


Registered in any course (%) 89.0 88.0 1.0 1.3 
Totalcreditsattempted = ss—i—i—sSS 1291 71 0.20 O49 
Totalcredisearned i ssts—<‘<i—<‘<—<‘<CSCSrdkK 1480002 001 | 0.20 
ReceivedaDorFinanycourse(%®) | 66 2} | 6.2 © 04 WZ 
Withdrew fromanycourse(%®) ~—~—~— -«19—~CS 7 86 24 15 
Sample size (total = 2,567) 1,284 1,283 


Cumulative semesters 


Total credits attempted 28.99 28.64 0.35 0.27 
Total credits earned 25.77 25.54 0.23 0.32 
R 1.9 


Withdrew from any course (%) : ; ; 1.9 


Sample size (total = 2,567) 1,284 1,283 


SOURCE: MDRC calculations using transcript data from the University of North Carolina at Charlotte. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by pre-random assignment characteristics of sample members. 
Spring and summer transcript records are combined. 
The second cohort is not included in the analysis of the second and cumulative semesters. 


Because the iPASS enhancements did not produce substantial contrasts in the early alerts or advis- 
ing support that students received, it is not surprising that the study enhancements did not lead to 
noticeable differences in short-term educational outcomes. 
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Conclusion 


" PASS is an ambitious initiative to integrate technology, data, and advising. Institutions must make 
choices about which kinds of technology to use, implement the new technologies, collect new 
data, help faculty members and advisers learn to use new systems, and synthesize everything into 

redesigned advising practices. The ultimate goal is to improve student outcomes through improved 
advising. So far, the enhancements undertaken by institutions in this study have not produced clear 
evidence of improvements in student outcomes. As described in the preceding chapters, this is largely 
because the enhancements generally produced only a modestly different student experience for stu- 
dents in the program group, compared with students in the control group. Rather than substantially 
altering how students experience college, the enhancements resulted in more incremental change. 


Even though the enhancements have not yet produced clear improvements in students’ academic 
performance, some staff members said they believe that their work in the iPASS initiative and their 
work on the enhancements studied here are important steps toward a stronger system to support 
students and help them succeed. It is clear that more work is needed: Across the three institutions, 
large proportions of students who were identified as being at high risk still earn Ds or Fs, or do not 
persist into subsequent semesters of college. 


The institutions in this project — California State University, Fresno; Montgomery County Community 
College; and the University of North Carolina at Charlotte — approached the study as a learning 
endeavor and an opportunity to make research-informed design decisions as they expand and en- 
hance iPASS practices. Each of the institutions made progress integrating technology and data with 
advising, getting more students in to see advisers, and expanding the content of advising sessions, 
but each also faced difficulties. In many ways their experiences highlight both the opportunities 
and the challenges that this kind of effort presents: 


e There is a tension between moving quickly to scale up advising reforms to all students and 
substantially changing students’ experiences. 


The reforms studied in this project were intended to integrate data and technology more fully into 
advising practices. Still, a large proportion of students never met with an adviser during the study 
period to experience how the reforms and enhancements could change advising sessions. The institu- 
tions also faced capacity challenges in implementing the reforms and enhancements for the subset 
of students in the study — for example, advisers did not always have time to learn how to use new 
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software, to use it with their students, or to meet with all of their students, key first steps in taking 
the enhanced reforms to a larger scale. The programs that are most effective at helping students 
stay on track to graduate substantially change the student experience, including multiple advising 
sessions per semester. The institutions in this study have experimented with ways to make advising 
tools and data accessible to students, advisers, and faculty. The next step is to build on this work to 
drive larger changes in the student experience. 


e Advisers would like richer advising sessions that go beyond course registration, but they gen- 
erally felt that they needed more time to meet with students. 


Even with new technologies and greater integration, advisers still say they do not have the capac- 
ity to effectively serve all their students. In fact, fully using the new data and technologies that 
were part of this project meant that advising sessions would need to be longer. At Fresno State and 
MCCC, the institutions used some of their grant resources to add staff members, which appeared to 
be an important factor in substantially increasing the number of students who had contact with an 
adviser. Using peer mentors, as Fresno State did, was less expensive than adding advisers, and once 
the practice was refined, it served to offload some of advisers’ work in a potentially cost-efficient 
way. Advisers reported that the toolbox that was used in the study helped them discuss topics with 
students in a more structured way and helped elicit more informative responses from students. 


¢ Registration holds can get more students to meet with advisers, but the details are important, 
as there may be unintended consequences. 


In this study, registration holds were designed to encourage students to meet with advisers during 
the semester, so that advisers could help with any issues students were facing and also help them plan 
for the subsequent semester. At Fresno State, these holds, coupled with a strong outreach strategy, 
substantially increased the number of students who had contact with advisers. At MCCC, however, 
students were not sufficiently motivated by the registration holds to meet with their advisers during 
the semester, and students may have taken fewer short-term courses later in the semester because 
of the holds. Approaches that work in one setting will not always work in another if the incentives 
and mandates are not aligned. 


¢ Collecting and responding to data about college students during the semester remains a 
challenge. 


Different ways of collecting data from students and faculty members could make integrated advising 
approaches more powerful, but collecting new information from students was especially difficult. 
Part of the design at some of the institutions included sending short surveys to students asking 
them about their classes and their academic progress. Survey responses were intended to inform 
the content of advising sessions with individual students, but few responded. Faculty members were 
better at responding and providing early-alert information about students who were exhibiting risk 
factors, although there was still inconsistency, and many faculty members did not respond. These 
types of early alerts could be automated for instructors who keep electronic grading systems up to 
date, which would facilitate real-time data collection and data-informed decision-making during 
advising sessions. 
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e Advanced data analytics presents opportunities but also risks, and simpler, more transparent 
solutions still work better in some cases. 


The two colleges that used proprietary predictive analytics technologies in this study experienced 
problems. At UNCC, staff members worked diligently to integrate data into advising sessions and 
to understand student risk factors, but proprietary software introduced errors into the risk assess- 
ments. At MCCC, a portion of students were identified as at risk who were actually performing well. 
At these colleges, advisers had less insight into the factors that identified at-risk students and less 
confidence in the assessments. Fresno State had the simplest approach, relying largely on students’ 
grade point averages rather than proprietary algorithms, and did not have a problem with risk as- 
sessment. In the second semester of the study, MCCC switched to using academic performance as 
the primary risk factor, based on feedback from advisers. UNCC also moved away from predictive 
analytics. The university used a graduation probability algorithm to identify the study sample for 
the first cohort of students but not the second. 


¢ Collaboration within the institutions allowed them to build on existing strengths. 


Staff members within each of the institutions collaborated to develop new practices to use existing 
technology and data, and to formalize those practices into a common set of guidelines that could be 
shared and used more broadly across the institutions. Each of the institutions demonstrated a com- 
mitment to strengthening student advising and support services. In developing the enhancements 
as part of this study, each institution established a foundation of resources, including standardized 
communication plans, messaging templates, a toolbox to guide advising sessions, and additional 
ways to help faculty members use early alerts. Together these resources can advance the use of 
technology and data in advising. 


WHAT’S NEXT 


The institutions in this study and across the iPASS initiative are making progress, and institutional 
practices are changing. As this report shows, however, it is still early, and there is still much to 
learn about how to use these new technologies and advising practices to produce improvements in 
student outcomes. 


This report has provided a closer look at how three institutions engaged in a rigorous process to en- 
hance and study their iPASS implementations. Although there is not yet evidence that these enhance- 
ments have improved student outcomes, the institutions’ work is ongoing. A subsequent report next 
year will present qualitative findings about the implementation of iPASS enhancements in greater 
detail, in addition to providing practical guidance for those interested in using new technologies 
to redesign their own advising practices. A final report will document the effects on longer-term 
student outcomes, using more data from all of the student cohorts included in this report. 
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Details of the Implementation Research 


The implementation research was guided by the following research questions: 


1. In what ways did implementation of the intervention components adhere to the intervention 
design? In what ways did the implementation not adhere to the intervention design? Why? 


2. How do the advising experiences of students in the intervention group compare with the experi- 
ences of students in the control group? 


To answer these questions, the research team conducted three-day site visits to each of the institutions 
during both the spring 2017 and fall 2017 semesters. During these site visits, researchers conducted 
individual interviews with a wide range of stakeholders, including students assigned to the program 
group, students assigned to the control group, peer mentors, advisers, iPASS project leaders, and 
other advising and senior administrators. Appendix Table A.1 provides additional details on the 
number of interviews conducted with stakeholders at each institution each semester. 


Interview protocols were designed to document implementation fidelity and degree of treatment 
contrast for each of the components of the interventions, including the application of registration 
holds, early-alert surveys, email and phone communications, and advising sessions. In addition, 
interview protocols were designed to document important contextual factors, including existing 
advising structures and technologies, capacity considerations related to advising, and other ongo- 
ing or recently implemented reforms that may have related to the institutions’ iPASS experiences. 


Typically, two researchers participated in an interview, with one researcher serving primarily as a 
note taker. If interviewees consented, interviews were audio recorded and transcribed. Researchers 
also took extensive field notes and prepared a detailed internal report summarizing initial impres- 
sions from each visit. 


In addition, the research team conducted phone interviews with advisers and iPASS project leaders 
at each institution in spring 2018. Protocols for these interviews were designed to document the 
institutions’ implementation experiences during the final semester of the intervention. In addition, 
these protocols asked about interviewees’ interest in retaining and scaling one or more components 
of the intervention and whether the institution had any plans to do so. 


The research team used Dedoose to code and analyze the data. Initially, the research team took an 
a priori coding approach, developing a codebook that aligned with the core components of the in- 
tervention and the iPASS grant.' A follow-up implementation report will draw on these data more 
extensively. 


1 Saldafia (2015). 


Studying Enhancements to Colleges’ iPASS Practices | 59 


APPENDIX TABLE A.1 Tally of Interviews with Stakeholders 


MONTGOMERY UNIVERSITY 


CALIFORNIA COUNTY OF NORTH 
STATE, COMMUNITY CAROLINA AT 

STAKEHOLDER FRESNO COLLEGE CHARLOTTE TOTAL 
Spring 2017 
Administrator 11 4 6 21 
iPASS projectleader ss stst—~S=S 2 ss fs. 
Advise 4 o  -° 6 aA 
Peermentr = ss ;  ., =. a ae 
Program group student = CS 7 1 Wn. 290 
Controlgroup student SS a ia... f 9 a1 
Fall 2017 
Administrator 6 4 4 14 
iPASSprojectleader i ti (iti kB HQ 
Adviser 2..." 4 6 430 
Peermentor | a ae a on 
Program group student == oo QO | io = 320 
Controlgroup student = ts—s—s=S .  . cs 7 OT 
Total 70 79 89 238 


SOURCE: Community College Research Center field research data. 
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Data Sources and Impact Estimation 
Model Specifications 


DATA SOURCES 


The technical analyses in this report rely on multiple data sources, described below. 


¢ MDRC used selected institutional-level measures from the Integrated Postsecondary Education 
Data System (IPEDS), housed at the U.S. Department of Education’s National Center for Education 
Statistics. The IPEDS data from the 2016-2017 academic year provide context for the iPASS study. 


e Student demographic data. The colleges provided MDRC with information on each student’s 
background and academic history. 


e iPASS program data. The colleges provided MDRC with information on sample members’ iPASS 
program activity, including early alerts and advising appointments, from their respective iPASS 
data systems. 


e Student transcript data. The colleges provided MDRC with information on the courses students 
took at their colleges, the grades they received, and credit hours for each course. 


THE EFFECT OF THE ENHANCED iPASS INTERVENTION 


The main analyses presented in Chapters 3, 4, and 5 are intent-to-treat estimates: The estimand, or 
the parameter being estimated, is the average treatment effect of being assigned to the enhanced 
iPASS group, regardless of whether students received the enhanced features of iPASS. This is not 
necessarily the same as the effect of experiencing enhanced iPASS, because, for example, not all 
students had contact with their advisers each semester. 


To conduct these analyses, a generalized least squares estimator is used to estimate the effect of the 
enhanced iPASS program at each of the three colleges. The following model specification is used: 


y=RTH+ >. Block; + > re +€ 


Here, y represents a target outcome, such as enrollment or credit accumulation. T is a binary indicator, 
equal to 1 if a student is randomly assigned to the program group and o otherwise. Block; is a vector 
of random assignment block indicators, equal to 1 if a student is in block j and 0 otherwise. Blocks 
are based on the unique college x adviser x cohort combination in which a student was randomly 
assigned. X, is a vector of baseline characteristics (listed below) that are included in the model to 
improve the precision of the estimates of B,.1 By is an estimator of the average effect of the intent 
to treat for the evaluation sample; weights are used so that this estimator is unbiased (described 
below). Missing indicators are included for those who are missing data on the baseline covariate. 
Baseline covariates include gender, race or ethnicity, and age for Fresno State and University of 
North Carolina at Charlotte impact estimates. Because Montgomery County Community College 
(MCCC) does not require a high school diploma or equivalency credential for enrollment, MCCC 


1 Bloom, Richburg-Hayes, and Black (2007). 
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impact estimates include an additional covariate that indicates whether a student had a high school 
diploma or General Educational Development (GED) certificate before enrolling. 


WEIGHTS FOR THE MCCC ANALYSES 


Weights are used in the main analyses for MCCC (Chapter 4) to account for the changing random 
assignment ratio by adviser. At the other institutions in this study, the random assignment ratio was 
set to 50:50 and random assignment was stratified by adviser. Because advisers had large student 
caseloads at MCCC, and program group students required more advising time and effort than control 
group students, the number of program group students for advisers at MCCC was capped at 45 per 
adviser during cohort 1 study enrollment. This meant that any adviser who was assigned more than 
go students would end up with 45 program group students and more than 45 control group students. 
No caps were applied during cohort 2 enrollment, since it is a smaller cohort and many cohort 1 
students did not re-enroll in the next semester. As a result, weights are calculated for each adviser 
block to account for resulting differences in the random assignment ratio. Weights are calculated to 
make the effective (weighted) random assignment ratio the same in all random assignment blocks 


p95) 4 py (t20 
wy = Pulp ( ij) =e, 


and equal to 50:50, as follows: 


where 


P, is equal to 1 if individual i in random assignment block j is assigned to the program group and o 
if the individual is assigned to the control group; and 


P; is equal to the proportion of sample members in random assignment block j assigned to the pro- 
gram group (that is, the average value of P,;, in random assignment block j). 
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ABOUT MDRC 


MDRC IS A NONPROFIT, NONPARTISAN SOCIAL AND EDU- 
CATION POLICY RESEARCH ORGANIZATION DEDICATED TO 
learning what works to improve the well-being of low-income 
people. Through its research and the active communication of its 
findings, MDRC seeks to enhance the effectiveness of social and 
education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; 
Washington, DC; and Los Angeles, MDRC is best known for 
mounting rigorous, large-scale, real-world tests of new and ex- 
isting policies and programs. Its projects are a mix of demon- 
strations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research 
and organizational experience to their work, providing expertise 
on the latest in qualitative and quantitative methods and on pro- 
gram design, development, implementation, and management. 
MDRC seeks to learn not just whether a program is effective but 
also how and why the program’s effects occur. In addition, it tries 
to place each project’s findings in the broader context of related 
research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, 
and best practices are shared with a broad audience in the policy 
and practitioner community as well as with the general public and 
the media. 


Over the years, MDRC has brought its unique approach to an 
ever-growing range of policy areas and target populations. 
Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, 
employment programs for ex-prisoners, and programs to help 
low-income students succeed in college. MDRC’s projects are 
organized into five areas: 


e Promoting Family Well-Being and Children’s Development 
e Improving Public Education 

e Raising Academic Achievement and Persistence in College 
e Supporting Low-Wage Workers and Communities 

e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, 
and Canada and the United Kingdom, MDRC conducts its proj- 
ects in partnership with national, state, and local governments, 
public school systems, community organizations, and numerous 
private philanthropies. 


